s3_object: NoSuchTagSet when s3 endpoint does not support tags #1607

dbarboza · 2023-06-08T17:39:19Z

Summary

As of Amazon AWS collection v6, the s3_object module fails with NoSuchTagSet when putting an object to an S3 endpoint that does not support object tagging. If I revert to 5.5.0, the same task works without error. Both 6.0.x and 6.1 exhibit the error. The reproduction task below works if I remove the Wasabi endpoint and run against AWS.

Issue Type

Bug Report

Component Name

s3_object

Ansible Version

ansible [core 2.15.0]
  config file = None
  configured module search path = ['/Users/dbarboza/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/dbarboza/Library/Python/3.11/lib/python/site-packages/ansible
  ansible collection location = /Users/dbarboza/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.11.3 (main, Apr  7 2023, 19:25:52) [Clang 14.0.0 (clang-1400.0.29.202)] (/usr/local/opt/[email protected]/bin/python3.11)
  jinja version = 3.1.2
  libyaml = True

Collection Versions

# /Users/dbarboza/.ansible/collections/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    6.1.0
ansible.eda                   1.3.8
ansible.netcommon             5.1.1
ansible.posix                 1.5.4
ansible.utils                 2.10.3
ansible.windows               1.14.0
arista.eos                    6.0.1
community.crypto              2.13.1
community.docker              3.4.6
community.general             7.0.1
community.grafana             1.5.4
dellemc.openmanage            7.6.1
kubernetes.core               2.4.0

# /Users/dbarboza/Library/Python/3.11/lib/python/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    6.0.1
ansible.netcommon             5.1.1
ansible.posix                 1.5.4
ansible.utils                 2.10.3
ansible.windows               1.14.0
arista.eos                    6.0.1
awx.awx                       22.2.0
azure.azcollection            1.15.0
check_point.mgmt              5.0.0
chocolatey.chocolatey         1.4.0
cisco.aci                     2.6.0
cisco.asa                     4.0.0
cisco.dnac                    6.7.2
cisco.intersight              1.0.27
cisco.ios                     4.5.0
cisco.iosxr                   5.0.2
cisco.ise                     2.5.12
cisco.meraki                  2.15.1
cisco.mso                     2.4.0
cisco.nso                     1.0.3
cisco.nxos                    4.3.0
cisco.ucs                     1.8.0
cloud.common                  2.1.3
cloudscale_ch.cloud           2.2.4
community.aws                 6.0.0
community.azure               2.0.0
community.ciscosmb            1.0.5
community.crypto              2.13.1
community.digitalocean        1.23.0
community.dns                 2.5.4
community.docker              3.4.6
community.fortios             1.0.0
community.general             7.0.1
community.google              1.0.0
community.grafana             1.5.4
community.hashi_vault         5.0.0
community.hrobot              1.8.0
community.libvirt             1.2.0
community.mongodb             1.5.2
community.mysql               3.7.1
community.network             5.0.0
community.okd                 2.3.0
community.postgresql          2.4.1
community.proxysql            1.5.1
community.rabbitmq            1.2.3
community.routeros            2.8.0
community.sap                 1.0.0
community.sap_libs            1.4.1
community.skydive             1.0.0
community.sops                1.6.1
community.vmware              3.6.0
community.windows             1.13.0

AWS SDK versions

Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: [email protected]
License: MIT
Location: /Users/dbarboza/Library/Python/3.11/lib/python/site-packages
Requires:
Required-by:
---
Name: boto3
Version: 1.26.149
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Users/dbarboza/Library/Python/3.11/lib/python/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.29.149
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Users/dbarboza/Library/Python/3.11/lib/python/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

CONFIG_FILE() = None
DEPRECATION_WARNINGS(env: ANSIBLE_DEPRECATION_WARNINGS) = True
PAGER(env: PAGER) = less

OS / Environment

MacOS Ventura 13.4

Steps to Reproduce

AWS_PROFILE=wasabi ansible localhost -m amazon.aws.s3_object -a "mode=put endpoint_url=https://s3.wasabisys.com bucket=bucket1 src=/tmp/foo object=foo " -vvv

Note that I changed the profile & bucket names for privacy.

Expected Results

I expected the object to be uploaded to the bucket.

Actual Results

AWS_PROFILE=wasabi ansible localhost -m amazon.aws.s3_object -a "mode=put endpoint_url=https://s3.wasabisys.com bucket=bucket1 src=/tmp/foo object=foo " -vvv
ansible [core 2.15.0]
  config file = None
  configured module search path = ['/Users/dbarboza/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/dbarboza/Library/Python/3.11/lib/python/site-packages/ansible
  ansible collection location = /Users/dbarboza/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.11.3 (main, Apr  7 2023, 19:25:52) [Clang 14.0.0 (clang-1400.0.29.202)] (/usr/local/opt/[email protected]/bin/python3.11)
  jinja version = 3.1.2
  libyaml = True
No config file found; using defaults
host_list declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
script declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
auto declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
yaml declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
ini declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
toml declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
[WARNING]: No inventory was parsed, only implicit localhost is available
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: dbarboza
<127.0.0.1> EXEC /bin/sh -c 'echo ~dbarboza && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/dbarboza/.ansible/tmp `"&& mkdir "` echo /Users/dbarboza/.ansible/tmp/ansible-tmp-1686243622.235886-25080-67772042048170 `" && echo ansible-tmp-1686243622.235886-25080-67772042048170="` echo /Users/dbarboza/.ansible/tmp/ansible-tmp-1686243622.235886-25080-67772042048170 `" ) && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'test -e /tmp/foo && sleep 0'
Using module file /Users/dbarboza/.ansible/collections/ansible_collections/amazon/aws/plugins/modules/s3_object.py
<127.0.0.1> PUT /Users/dbarboza/.ansible/tmp/ansible-local-250430uz258vj/tmpky7sks1f TO /Users/dbarboza/.ansible/tmp/ansible-tmp-1686243622.235886-25080-67772042048170/AnsiballZ_s3_object.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/dbarboza/.ansible/tmp/ansible-tmp-1686243622.235886-25080-67772042048170/ /Users/dbarboza/.ansible/tmp/ansible-tmp-1686243622.235886-25080-67772042048170/AnsiballZ_s3_object.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/local/opt/[email protected]/bin/python3.11 /Users/dbarboza/.ansible/tmp/ansible-tmp-1686243622.235886-25080-67772042048170/AnsiballZ_s3_object.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /Users/dbarboza/.ansible/tmp/ansible-tmp-1686243622.235886-25080-67772042048170/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/var/folders/rl/2llc0bfj09v32zs7dlr9n0hr0000gn/T/ansible_amazon.aws.s3_object_payload_q_z8lkwy/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 1026, in ensure_tags
  File "/var/folders/rl/2llc0bfj09v32zs7dlr9n0hr0000gn/T/ansible_amazon.aws.s3_object_payload_q_z8lkwy/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 974, in get_current_object_tags_dict
  File "/var/folders/rl/2llc0bfj09v32zs7dlr9n0hr0000gn/T/ansible_amazon.aws.s3_object_payload_q_z8lkwy/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/retries.py", line 105, in deciding_wrapper
    return retrying_wrapper(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/rl/2llc0bfj09v32zs7dlr9n0hr0000gn/T/ansible_amazon.aws.s3_object_payload_q_z8lkwy/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 119, in _retry_wrapper
    return _retry_func(
           ^^^^^^^^^^^^
  File "/var/folders/rl/2llc0bfj09v32zs7dlr9n0hr0000gn/T/ansible_amazon.aws.s3_object_payload_q_z8lkwy/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 68, in _retry_func
    return func()
           ^^^^^^
  File "/Users/dbarboza/Library/Python/3.11/lib/python/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dbarboza/Library/Python/3.11/lib/python/site-packages/botocore/client.py", line 964, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (NoSuchTagSet) when calling the GetObjectTagging operation: The TagSet does not exist

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/folders/rl/2llc0bfj09v32zs7dlr9n0hr0000gn/T/ansible_amazon.aws.s3_object_payload_q_z8lkwy/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 1540, in main
  File "/var/folders/rl/2llc0bfj09v32zs7dlr9n0hr0000gn/T/ansible_amazon.aws.s3_object_payload_q_z8lkwy/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 1188, in s3_object_do_put
  File "/var/folders/rl/2llc0bfj09v32zs7dlr9n0hr0000gn/T/ansible_amazon.aws.s3_object_payload_q_z8lkwy/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 1032, in ensure_tags
S3ObjectFailure: Failed to get object tags.
localhost | FAILED! => {
    "boto3_version": "1.26.149",
    "botocore_version": "1.29.149",
    "changed": false,
    "error": {
        "code": "NoSuchTagSet",
        "message": "The TagSet does not exist"
    },
    "invocation": {
        "module_args": {
            "access_key": null,
            "aws_ca_bundle": null,
            "aws_config": null,
            "bucket": "bucket1",
            "ceph": false,
            "content": null,
            "content_base64": null,
            "copy_src": null,
            "debug_botocore_endpoint_logs": false,
            "dest": null,
            "dualstack": false,
            "encrypt": true,
            "encryption_kms_key_id": null,
            "encryption_mode": "AES256",
            "endpoint_url": "https://s3.wasabisys.com",
            "expiry": 600,
            "headers": null,
            "ignore_nonexistent_bucket": false,
            "marker": "",
            "max_keys": 1000,
            "metadata": null,
            "mode": "put",
            "object": "foo",
            "overwrite": "different",
            "permission": [
                "private"
            ],
            "prefix": "",
            "profile": "wasabi",
            "purge_tags": true,
            "region": null,
            "retries": 0,
            "secret_key": null,
            "session_token": null,
            "sig_v4": true,
            "src": "/tmp/foo",
            "tags": null,
            "validate_bucket_name": true,
            "validate_certs": true,
            "version": null
        }
    },
    "msg": "Failed to get object tags.: An error occurred (NoSuchTagSet) when calling the GetObjectTagging operation: The TagSet does not exist",
    "response_metadata": {
        "host_id": "ULVVsAhNyIYClSGUxricoaonhDPqHecfHckAwQ2fAWsWHyEqz7css1vdbEPUXIriCSY+O6vsP4av",
        "http_headers": {
            "content-type": "application/xml",
            "date": "Thu, 08 Jun 2023 17:00:23 GMT",
            "server": "WasabiS3/7.14.291-2023-05-15-3fb36ca10e (head12)",
            "transfer-encoding": "chunked",
            "x-amz-id-2": "ULVVsAhNyIYClSGUxricoaonhDPqHecfHckAwQ2fAWsWHyEqz7css1vdbEPUXIriCSY+O6vsP4av",
            "x-amz-request-id": "8E8620150D440987"
        },
        "http_status_code": 404,
        "request_id": "8E8620150D440987",
        "retry_attempts": 0
    }
}

Code of Conduct

I agree to follow the Ansible Code of Conduct

The text was updated successfully, but these errors were encountered:

ansibullbot · 2023-07-12T17:45:32Z

Files identified in the description:

[plugins/modules/s3_object.py](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/modules/s3_object.py)

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

ansibullbot · 2023-07-12T17:45:33Z

cc @alinabuzachis @jillr @lwade @s-hertel @tremble
click here for bot help

…g bucket tags

…ags (#1813) s3_object - fix NoSuchTagSet error when S3 endpoint doesn't support tags SUMMARY is_boto3_error_code only supports being passed a string or list. In the refactor of s3_object it was passed a tuple, so the errors weren't being cleanly caught. ISSUE TYPE Bugfix Pull Request COMPONENT NAME s3_object ADDITIONAL INFORMATION fixes: #1607 Reviewed-by: Alina Buzachis

…ags (#1813) s3_object - fix NoSuchTagSet error when S3 endpoint doesn't support tags SUMMARY is_boto3_error_code only supports being passed a string or list. In the refactor of s3_object it was passed a tuple, so the errors weren't being cleanly caught. ISSUE TYPE Bugfix Pull Request COMPONENT NAME s3_object ADDITIONAL INFORMATION fixes: #1607 Reviewed-by: Alina Buzachis (cherry picked from commit 985d8b1)

…ags (#1813) (#1814) [PR #1813/985d8b1c backport][stable-6] s3_object - fix NoSuchTagSet error when S3 endpoint doesn't support tags This is a backport of PR #1813 as merged into main (985d8b1). SUMMARY is_boto3_error_code only supports being passed a string or list. In the refactor of s3_object it was passed a tuple, so the errors weren't being cleanly caught. ISSUE TYPE Bugfix Pull Request COMPONENT NAME s3_object ADDITIONAL INFORMATION fixes: #1607 Reviewed-by: Mark Chappell

Update contributing to sync with amazon.aws Fixes: ansible-collections#1607 SUMMARY Update CONTRIBUTING.md to sync with amazon.aws and pick up various recent improvements (See also ansible-collections#924) ISSUE TYPE Docs Pull Request COMPONENT NAME CONTRIBUTING.md ADDITIONAL INFORMATION Reviewed-by: Alina Buzachis

github-actions bot added the needs_triage label Jun 8, 2023

alinabuzachis added needs_verified Some one might want to take a look at this and reproduce it to confirm jira and removed needs_triage labels Jun 13, 2023

ansibullbot added bug This issue/PR relates to a bug module module python3 traceback labels Jul 12, 2023

bingopriya added a commit to bingopriya/amazon.aws that referenced this issue Jul 27, 2023

Fixes ansible-collections#1607 - Catch botocore exception when gettin…

3b8ae44

…g bucket tags

bingopriya mentioned this issue Jul 27, 2023

Fixes #1607 - Catch botocore exception when getting bucket tags #1667

Closed

tremble mentioned this issue Oct 19, 2023

s3_object - fix NoSuchTagSet error when S3 endpoint doesn't support tags #1813

Merged

softwarefactory-project-zuul bot closed this as completed in #1813 Oct 19, 2023

patchback bot mentioned this issue Oct 19, 2023

[PR #1813/985d8b1c backport][stable-6] s3_object - fix NoSuchTagSet error when S3 endpoint doesn't support tags #1814

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

s3_object: NoSuchTagSet when s3 endpoint does not support tags #1607

s3_object: NoSuchTagSet when s3 endpoint does not support tags #1607

dbarboza commented Jun 8, 2023 •

edited

Loading

ansibullbot commented Jul 12, 2023

ansibullbot commented Jul 12, 2023

s3_object: NoSuchTagSet when s3 endpoint does not support tags #1607

s3_object: NoSuchTagSet when s3 endpoint does not support tags #1607

Comments

dbarboza commented Jun 8, 2023 • edited Loading

Summary

Issue Type

Component Name

Ansible Version

Collection Versions

AWS SDK versions

Configuration

OS / Environment

Steps to Reproduce

Expected Results

Actual Results

Code of Conduct

ansibullbot commented Jul 12, 2023

ansibullbot commented Jul 12, 2023

dbarboza commented Jun 8, 2023 •

edited

Loading