Announcement: S3 default integrity change

[RyanFitzSimmonsAK]

In AWS SDK for Python v1.36.0, we released changes to the S3 client that adopts new default integrity protections. For more information on default integrity behavior, please refer to the official SDK documentation. In SDK releases from this version on, clients default to enabling an additional checksum on all Put calls and enabling validation on Get calls.
You can disable default integrity protections for S3. We do not recommend this because checksums are important to S3 integrity posture. Integrity protections can be disabled by setting the config flag to when_required, or by using the related AWS shared config file settings or environment variables.

Disclaimer: The AWS SDKs and CLI are designed for usage with official AWS services. We may introduce and enable new features by default, such as these new default integrity protections prior to them being supported or handled by third-party service implementations. You can disable the new behavior with the WHEN_REQUIRED value for the request_checksum_calculation and response_checksum_validation configuration options covered in Data Integrity Protections for Amazon S3.

[violuke]

Surely this should have been a breaking change?

[NGPixel]

They did the same thing in the JS client. Leave it to AWS to introduce breaking changes and pretend their library isn't used extensively with other services...

[jeking3]

Ouch.. this really should have been a major version bump or opt-in instead of opt-out.

[FChmiel]

+1

ImportError: cannot import name 'DEFAULT_CHECKSUM_ALGORITHM' from 'botocore.httpchecksum'

[jonathan343]

For those using third-party service implementations, please see the updated disclaimer above:

Disclaimer: The AWS SDKs and CLI are designed for usage with official AWS services. We may introduce and enable new features by default, such as these new default integrity protections prior to them being supported or handled by third-party service implementations. You can disable the new behavior with the WHEN_REQUIRED value for the request_checksum_calculation and response_checksum_validation configuration options covered in Data Integrity Protections for Amazon S3.

[jonathan343]

ImportError: cannot import name 'DEFAULT_CHECKSUM_ALGORITHM' from 'botocore.httpchecksum'

@FChmiel this is most likely a result of not using the latest version of botocore where this constant was introduced (botocore-1.36.0). This is the minimum supported version for boto3-1.36.0, however, if you're using the s3transfer package without boto3, we've identified an relevant issue and merged a fix for this last night which will be included in today's daily release for s3transfer-0.11.1.

Tracking issue: boto/s3transfer#324

[talsalmona]

Just so you know - this completely broke the sns_extended_client. sns_extended_client is an AWS implementation.

[IsaevIlya]

hey @jonathan343,
We're encountering the same issue with our integration tests running in the cibuildwheel container. I've reviewed the logs, and we consistently install the latest boto3/botocore version during each test run. This suggests that the issue isn't related to using an outdated version of botocore, as you had proposed. For reference, here are the relevant version details from our logs:

cibuildwheel version 2.22.0

Build options:
  platform: linux
  allow_empty: False
  architectures: aarch64, armv7l, i686, ppc64le, s390x, x86_64
  build_selector: 
    build_config: cp39-manylinux_aarch64
    skip_config: 
    requires_python: <3.13,>=3.8
    enable: ['cpython-freethreading', 'cpython-prerelease', 'pypy']

    Collecting boto3 (from s3torchconnectorclient==1.3.1->s3torchconnectorclient==1.3.1)
    Downloading boto3-1.36.0-py3-none-any.whl.metadata (6.6 kB)
    ...
    Collecting botocore<1.37.0,>=1.36.0 (from boto3->s3torchconnectorclient==1.3.1->s3torchconnectorclient==1.3.1)
    Downloading botocore-1.36.0-py3-none-any.whl.metadata (5.7 kB)

[FChmiel]

Thanks @jonathan343.

I also suspect its not us using an outdated botocore version: We only pin the boto3 version and botocore is a transitive dependancy of this only.

[jonathan343]

@IsaevIlya @FChmiel Are either of you able to provide a reproducible error? Also, can you verify the correct versions in your environment after installing all dependencies using something like python -m pip freeze?

The following shows me installing the latest version of boto3 from PyPI and successfully importing DEFAULT_CHECKSUM_ALGORITHM from botocore.httpchecksum.

$ python -m venv venv
$ . venv/bin/activate 
(venv) $ pip install boto3 --no-cache-dir 
Collecting boto3
  Downloading boto3-1.36.0-py3-none-any.whl.metadata (6.6 kB)
Collecting botocore<1.37.0,>=1.36.0 (from boto3)
  Downloading botocore-1.36.0-py3-none-any.whl.metadata (5.7 kB)
Collecting jmespath<2.0.0,>=0.7.1 (from boto3)
  Downloading jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB)
Collecting s3transfer<0.12.0,>=0.11.0 (from boto3)
  Downloading s3transfer-0.11.0-py3-none-any.whl.metadata (1.7 kB)
Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.37.0,>=1.36.0->boto3)
  Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting urllib3!=2.2.0,<3,>=1.25.4 (from botocore<1.37.0,>=1.36.0->boto3)
  Downloading urllib3-2.3.0-py3-none-any.whl.metadata (6.5 kB)
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.37.0,>=1.36.0->boto3)
  Downloading six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Downloading boto3-1.36.0-py3-none-any.whl (139 kB)
Downloading botocore-1.36.0-py3-none-any.whl (13.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.3/13.3 MB 111.8 MB/s eta 0:00:00
Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Downloading s3transfer-0.11.0-py3-none-any.whl (84 kB)
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Downloading urllib3-2.3.0-py3-none-any.whl (128 kB)
Downloading six-1.17.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: urllib3, six, jmespath, python-dateutil, botocore, s3transfer, boto3
Successfully installed boto3-1.36.0 botocore-1.36.0 jmespath-1.0.1 python-dateutil-2.9.0.post0 s3transfer-0.11.0 six-1.17.0 urllib3-2.3.0
(venv) $ pip freeze
boto3==1.36.0
botocore==1.36.0
jmespath==1.0.1
python-dateutil==2.9.0.post0
s3transfer==0.11.0
six==1.17.0
urllib3==2.3.0
(venv) $ python
Python 3.13.1 (main, Dec 12 2024, 00:56:51) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from botocore.httpchecksum import DEFAULT_CHECKSUM_ALGORITHM
>>> DEFAULT_CHECKSUM_ALGORITHM
'CRC32'

However, when I install a version of botocore < 1.36.0 in the same environment (this is a dependency conflict) I get the same ImportError you're seeing:

(venv) $ pip install botocore==1.35.99
Collecting botocore==1.35.99
  Using cached botocore-1.35.99-py3-none-any.whl.metadata (5.7 kB)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in ./venv/lib/python3.13/site-packages (from botocore==1.35.99) (1.0.1)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in ./venv/lib/python3.13/site-packages (from botocore==1.35.99) (2.9.0.post0)
Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in ./venv/lib/python3.13/site-packages (from botocore==1.35.99) (2.3.0)
Requirement already satisfied: six>=1.5 in ./venv/lib/python3.13/site-packages (from python-dateutil<3.0.0,>=2.1->botocore==1.35.99) (1.17.0)
Using cached botocore-1.35.99-py3-none-any.whl (13.3 MB)
Installing collected packages: botocore
  Attempting uninstall: botocore
    Found existing installation: botocore 1.36.0
    Uninstalling botocore-1.36.0:
      Successfully uninstalled botocore-1.36.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
boto3 1.36.0 requires botocore<1.37.0,>=1.36.0, but you have botocore 1.35.99 which is incompatible.
Successfully installed botocore-1.35.99
(venv) $ pip freeze                      
boto3==1.36.0
botocore==1.35.99
jmespath==1.0.1
python-dateutil==2.9.0.post0
s3transfer==0.11.0
six==1.17.0
urllib3==2.3.0
(venv) $ python                       
Python 3.13.1 (main, Dec 12 2024, 00:56:51) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from botocore.httpchecksum import DEFAULT_CHECKSUM_ALGORITHM
Traceback (most recent call last):
  File "<python-input-0>", line 1, in <module>
    from botocore.httpchecksum import DEFAULT_CHECKSUM_ALGORITHM
ImportError: cannot import name 'DEFAULT_CHECKSUM_ALGORITHM' from 'botocore.httpchecksum' (/Users/gytndd/Desktop/oncall/venv/lib/python3.13/site-packages/botocore/httpchecksum.py)

[FChmiel]

Thanks, I replicate your (correct) behaviour with poetry when trying to generate a a reproducible error.

In my lock file I have s3transfer==0.11.0, so that may be it.

[kdaily]

@talsalmona - thanks for the report. I've sent the concern to the maintainers of that package. If you can, it would be helpful to provide more details on the failure scenario on the amazon-sns-python-extended-client-lib repository:

https://github.com/awslabs/amazon-sns-python-extended-client-lib/issues

[talsalmona]

@talsalmona - thanks for the report. I've sent the concern to the maintainers of that package. If you can, it would be helpful to provide more details on the failure scenario on the amazon-sns-python-extended-client-lib repository:

https://github.com/awslabs/amazon-sns-python-extended-client-lib/issues

Sure, added here: awslabs/amazon-sns-python-extended-client-lib#20

[IsaevIlya]

Hey @jonathan343,
I'm unable to reproduce the issue anymore. Today our integration tests are picking new versions with which everything working fine. I suppose whatever issue was, it was resolved for us:

boto3==1.36.1
botocore==1.36.1
...
s3transfer==0.11.1