s3_object put with metadata specifying Content-Type results in ContentType being None (and boto3 raising exception)

[timdiggins]

Summary

If I want to upload a file with a specific content-type, then I will set the metadata to have Content-Type:

  - name: upload json
    amazon.aws.s3_object:
      bucket: "{{ discovery_bucket_name }}"
      content: "{{ discovery_tempfile.stdout }}"
      object: "discovery.json"
      mode: put
      metadata: 'ContentType=application/json'

however this causes the following kind of error:

Traceback

Traceback (most recent call last):
  File "/tmp/ansible_amazon.aws.s3_object_payload_l1xlmt9s/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 756, in upload_s3file
  File "/tmp/ansible_amazon.aws.s3_object_payload_l1xlmt9s/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/retries.py", line 105, in deciding_wrapper
    return retrying_wrapper(*args, **kwargs)
  File "/tmp/ansible_amazon.aws.s3_object_payload_l1xlmt9s/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 119, in _retry_wrapper
    return _retry_func(
  File "/tmp/ansible_amazon.aws.s3_object_payload_l1xlmt9s/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 68, in _retry_func
    return func()
  File "/usr/local/lib/python3.9/site-packages/boto3/s3/inject.py", line 636, in upload_fileobj
    return future.result()
  File "/usr/local/lib/python3.9/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/usr/local/lib/python3.9/site-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/s3transfer/tasks.py", line 139, in __call__
    return self._execute_main(kwargs)
  File "/usr/local/lib/python3.9/site-packages/s3transfer/tasks.py", line 162, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/s3transfer/upload.py", line 764, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 535, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 936, in _make_api_call
    request_dict = self._convert_to_request_dict(
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 1010, in _convert_to_request_dict
    request_dict = self._serializer.serialize_to_request(
  File "/usr/local/lib/python3.9/site-packages/botocore/validate.py", line 381, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter ContentType, value: None, type: <class 'NoneType'>, valid types: <class 'str'>

This seems to be because this following code:

https://github.com/ansible-collections/amazon.aws/blob/main/plugins/modules/s3_object.py#L750

replaces extra["ContentType"] with the return value of get_content_type which, when present = truthy implicitly returns None:

https://github.com/ansible-collections/amazon.aws/blob/main/plugins/modules/s3_object.py#L657-L665

This was introduced in #1193 -- I have added the following comment: https://github.com/ansible-collections/amazon.aws/pull/1193/files#r1406429232

Issue Type

Bug Report

Component Name

s3_object

Ansible Version

$ ansible --version

ansible [core 2.13.13]
  config file = /canupload/provision/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.18 (main, Nov 21 2023, 19:14:17) [GCC 10.2.1 20210110]
  jinja version = 3.1.2
  libyaml = True

Collection Versions

$ ansible-galaxy collection list


# /usr/local/lib/python3.9/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    3.5.0
ansible.netcommon             3.1.3
ansible.posix                 1.4.0
ansible.utils                 2.8.0
ansible.windows               1.12.0
arista.eos                    5.0.1
awx.awx                       21.10.0
azure.azcollection            1.14.0
check_point.mgmt              2.3.0
chocolatey.chocolatey         1.3.1
cisco.aci                     2.3.0
cisco.asa                     3.1.0
cisco.dnac                    6.6.1
cisco.intersight              1.0.22
cisco.ios                     3.3.2
cisco.iosxr                   3.3.1
cisco.ise                     2.5.9
cisco.meraki                  2.13.0
cisco.mso                     2.1.0
cisco.nso                     1.0.3
cisco.nxos                    3.2.0
cisco.ucs                     1.8.0
cloud.common                  2.1.2
cloudscale_ch.cloud           2.2.3
community.aws                 3.6.0
community.azure               1.1.0
community.ciscosmb            1.0.5
community.crypto              2.9.0
community.digitalocean        1.22.0
community.dns                 2.4.2
community.docker              2.7.3
community.fortios             1.0.0
community.general             5.8.3
community.google              1.0.0
community.grafana             1.5.3
community.hashi_vault         3.4.0
community.hrobot              1.6.0
community.libvirt             1.2.0
community.mongodb             1.4.2
community.mysql               3.5.1
community.network             4.0.2
community.okd                 2.2.0
community.postgresql          2.3.1
community.proxysql            1.4.0
community.rabbitmq            1.2.3
community.routeros            2.5.0
community.sap                 1.0.0
community.sap_libs            1.4.0
community.skydive             1.0.0
community.sops                1.5.0
community.vmware              2.10.2
community.windows             1.11.1
community.zabbix              1.9.0
containers.podman             1.10.1
cyberark.conjur               1.2.0
cyberark.pas                  1.0.14
dellemc.enterprise_sonic      1.1.2
dellemc.openmanage            5.5.0
dellemc.os10                  1.1.1
dellemc.os6                   1.0.7
dellemc.os9                   1.0.4
f5networks.f5_modules         1.21.0
fortinet.fortimanager         2.1.7
fortinet.fortios              2.2.1
frr.frr                       2.0.0
gluster.gluster               1.0.2
google.cloud                  1.0.2
hetzner.hcloud                1.9.0
hpe.nimble                    1.1.4
ibm.qradar                    2.1.0
ibm.spectrum_virtualize       1.10.0
infinidat.infinibox           1.3.12
infoblox.nios_modules         1.4.1
inspur.ispim                  1.2.0
inspur.sm                     2.3.0
junipernetworks.junos         3.1.0
kubernetes.core               2.3.2
lowlydba.sqlserver            1.2.0
mellanox.onyx                 1.0.0
netapp.aws                    21.7.0
netapp.azure                  21.10.0
netapp.cloudmanager           21.21.0
netapp.elementsw              21.7.0
netapp.ontap                  21.24.1
netapp.storagegrid            21.11.1
netapp.um_info                21.8.0
netapp_eseries.santricity     1.3.1
netbox.netbox                 3.9.0
ngine_io.cloudstack           2.3.0
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.2
openstack.cloud               1.10.0
openvswitch.openvswitch       2.1.0
ovirt.ovirt                   2.4.1
purestorage.flasharray        1.15.0
purestorage.flashblade        1.10.0
purestorage.fusion            1.2.0
sensu.sensu_go                1.13.1
servicenow.servicenow         1.0.6
splunk.es                     2.1.0
t_systems_mms.icinga_director 1.31.4
theforeman.foreman            3.7.0
vmware.vmware_rest            2.2.0
vultr.cloud                   1.3.1
vyos.vyos                     3.0.1
wti.remote                    1.0.4

# /root/.ansible/collections/ansible_collections
Collection             Version
---------------------- -------
amazon.aws             7.0.0
community.aws          7.0.0
community.digitalocean 1.24.0
vultr.cloud            1.10.0

AWS SDK versions

$ pip3 show boto boto3 botocore

pip3 show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.29.6
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /usr/local/lib/python3.9/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.32.6
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /usr/local/lib/python3.9/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed

OS / Environment

No response

Steps to Reproduce

  - name: upload json
    amazon.aws.s3_object:
      bucket: "{{ discovery_bucket_name }}"
      content: "{{ discovery_tempfile.stdout }}"
      object: "discovery.json"
      mode: put
      metadata: 'ContentType=application/json'

Expected Results

I expected the file to be uploaded with a content type of application/json

Actual Results

Error in Boto3 because ContentType is passed as None

Traceback (most recent call last):
  File "/tmp/ansible_amazon.aws.s3_object_payload_l1xlmt9s/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 756, in upload_s3file
  File "/tmp/ansible_amazon.aws.s3_object_payload_l1xlmt9s/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/retries.py", line 105, in deciding_wrapper
    return retrying_wrapper(*args, **kwargs)
  File "/tmp/ansible_amazon.aws.s3_object_payload_l1xlmt9s/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 119, in _retry_wrapper
    return _retry_func(
  File "/tmp/ansible_amazon.aws.s3_object_payload_l1xlmt9s/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 68, in _retry_func
    return func()
  File "/usr/local/lib/python3.9/site-packages/boto3/s3/inject.py", line 636, in upload_fileobj
    return future.result()
  File "/usr/local/lib/python3.9/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/usr/local/lib/python3.9/site-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/s3transfer/tasks.py", line 139, in __call__
    return self._execute_main(kwargs)
  File "/usr/local/lib/python3.9/site-packages/s3transfer/tasks.py", line 162, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/s3transfer/upload.py", line 764, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 535, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 936, in _make_api_call
    request_dict = self._convert_to_request_dict(
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 1010, in _convert_to_request_dict
    request_dict = self._serializer.serialize_to_request(
  File "/usr/local/lib/python3.9/site-packages/botocore/validate.py", line 381, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter ContentType, value: None, type: <class 'NoneType'>, valid types: <class 'str'>

Code of Conduct

• I agree to follow the Ansible Code of Conduct