aws_ssm connection plugin: S3 Signed Url invalid for newly created S3 Bucket #637
Comments
|
Files identified in the description:
If these files are inaccurate, please update the |
|
Closing this temporarily as I think I need to revisit some of this first, apologies! To be more specific; the errors are still happening, but my determination of the cause may not be right, and I want to dig into that more before someone else starts looking. EDIT: I've reopened the issue now, as I've realised that I was just confusing myself a bit reading the fix I'd hacked together myself. The issue is still present when using the latest |
|
Files identified in the description:
If these files are inaccurate, please update the |
|
@garethsaxby thx for the detailed report! Sadly I'm not familiar with ssm connection plugin.
There is a similar PR (not merged yet) that introduce a if bucket_region:
client = session.client(
service,
config=Config(signature_version="s3v4"),
endpoint_url=f'https://s3.{bucket_region}.amazonaws.com'
)
else:
client = session.client(
service,
config=Config(signature_version="s3v4")
) |
|
I also confirm this issue. I think a issue fix would be to use '-L' option with curl. |
|
I'm definitely having issues with bucket regions and encrypted buckets too. |
…n var (#1176) SUMMARY Fix issue where syntax error is reported if using ssm connection and the target node is located in a different region to the s3 bucket. Fixes #1190, #637 ISSUE TYPE Bugfix Pull Request COMPONENT NAME aws_ssm connection plugin ADDITIONAL INFORMATION When using SSM for ansible connection and the target node is in a different region to the s3 bucket used, the playbook immediately errors with the following. (There are no issues when both target and s3 bucket are in the same region) fatal: [i-04444a7f03cc2bffd]: FAILED! => {"ansible_facts": {}, "changed": false, "failed_modules": {"ansible.legacy.setup": {"failed": true, "module_stderr": "", "module_stdout": " File \"/tmp/ansible/ansible-tmp-1653576081.8378458-29658-258097978113216/AnsiballZ_setup.py\", line 1\r\r\n <?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\r\n ^\r\r\nSyntaxError: invalid syntax\r\r", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}}, "msg": "The following modules failed to execute: ansible.legacy.setup\n"} The tmp file has the following contents <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AuthorizationQueryParametersError</Code><Message>Error parsing the X-Amz-Credential parameter; the region 'us-east-1' is wrong; expecting 'eu-west-1'</Message><Region>eu-west-1</Region><RequestId>4VTAGR4C1V9ATBJT</RequestId><HostId>OahjGsFQHlr3ihxobH/yyH7Mzxq98mwjcb6+J3Y2EifDU7FykCe8b6QJTNodIG5WSquVeJF+Zsk=</HostId></Error> Steps to reproduce: run an ansible playbook using the following: aws_ec2 inventory aws_ssm connection type specify an s3 bucket (ansible_aws_ssm_bucket_name var) that is in a different location to the target node The presigned url generated includes the region the s3 bucket is in, so this region must be used for the session obtained in the _get_url function.
…n var (#1176) SUMMARY Fix issue where syntax error is reported if using ssm connection and the target node is located in a different region to the s3 bucket. Fixes #1190, #637 ISSUE TYPE Bugfix Pull Request COMPONENT NAME aws_ssm connection plugin ADDITIONAL INFORMATION When using SSM for ansible connection and the target node is in a different region to the s3 bucket used, the playbook immediately errors with the following. (There are no issues when both target and s3 bucket are in the same region) fatal: [i-04444a7f03cc2bffd]: FAILED! => {"ansible_facts": {}, "changed": false, "failed_modules": {"ansible.legacy.setup": {"failed": true, "module_stderr": "", "module_stdout": " File \"/tmp/ansible/ansible-tmp-1653576081.8378458-29658-258097978113216/AnsiballZ_setup.py\", line 1\r\r\n <?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\r\n ^\r\r\nSyntaxError: invalid syntax\r\r", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}}, "msg": "The following modules failed to execute: ansible.legacy.setup\n"} The tmp file has the following contents <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AuthorizationQueryParametersError</Code><Message>Error parsing the X-Amz-Credential parameter; the region 'us-east-1' is wrong; expecting 'eu-west-1'</Message><Region>eu-west-1</Region><RequestId>4VTAGR4C1V9ATBJT</RequestId><HostId>OahjGsFQHlr3ihxobH/yyH7Mzxq98mwjcb6+J3Y2EifDU7FykCe8b6QJTNodIG5WSquVeJF+Zsk=</HostId></Error> Steps to reproduce: run an ansible playbook using the following: aws_ec2 inventory aws_ssm connection type specify an s3 bucket (ansible_aws_ssm_bucket_name var) that is in a different location to the target node The presigned url generated includes the region the s3 bucket is in, so this region must be used for the session obtained in the _get_url function. (cherry picked from commit 1be7da1)
…n var (#1176) SUMMARY Fix issue where syntax error is reported if using ssm connection and the target node is located in a different region to the s3 bucket. Fixes #1190, #637 ISSUE TYPE Bugfix Pull Request COMPONENT NAME aws_ssm connection plugin ADDITIONAL INFORMATION When using SSM for ansible connection and the target node is in a different region to the s3 bucket used, the playbook immediately errors with the following. (There are no issues when both target and s3 bucket are in the same region) fatal: [i-04444a7f03cc2bffd]: FAILED! => {"ansible_facts": {}, "changed": false, "failed_modules": {"ansible.legacy.setup": {"failed": true, "module_stderr": "", "module_stdout": " File \"/tmp/ansible/ansible-tmp-1653576081.8378458-29658-258097978113216/AnsiballZ_setup.py\", line 1\r\r\n <?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\r\n ^\r\r\nSyntaxError: invalid syntax\r\r", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}}, "msg": "The following modules failed to execute: ansible.legacy.setup\n"} The tmp file has the following contents <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AuthorizationQueryParametersError</Code><Message>Error parsing the X-Amz-Credential parameter; the region 'us-east-1' is wrong; expecting 'eu-west-1'</Message><Region>eu-west-1</Region><RequestId>4VTAGR4C1V9ATBJT</RequestId><HostId>OahjGsFQHlr3ihxobH/yyH7Mzxq98mwjcb6+J3Y2EifDU7FykCe8b6QJTNodIG5WSquVeJF+Zsk=</HostId></Error> Steps to reproduce: run an ansible playbook using the following: aws_ec2 inventory aws_ssm connection type specify an s3 bucket (ansible_aws_ssm_bucket_name var) that is in a different location to the target node The presigned url generated includes the region the s3 bucket is in, so this region must be used for the session obtained in the _get_url function. (cherry picked from commit 1be7da1)
…n var (#1176) (#1291) [PR #1176/1be7da11 backport][stable-4] ssm connection: pull bucket region info rather than taking from region var This is a backport of PR #1176 as merged into main (1be7da1). SUMMARY Fix issue where syntax error is reported if using ssm connection and the target node is located in a different region to the s3 bucket. Fixes #1190, #637 ISSUE TYPE Bugfix Pull Request COMPONENT NAME aws_ssm connection plugin ADDITIONAL INFORMATION When using SSM for ansible connection and the target node is in a different region to the s3 bucket used, the playbook immediately errors with the following. (There are no issues when both target and s3 bucket are in the same region) fatal: [i-04444a7f03cc2bffd]: FAILED! => {"ansible_facts": {}, "changed": false, "failed_modules": {"ansible.legacy.setup": {"failed": true, "module_stderr": "", "module_stdout": " File \"/tmp/ansible/ansible-tmp-1653576081.8378458-29658-258097978113216/AnsiballZ_setup.py\", line 1\r\r\n <?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\r\n ^\r\r\nSyntaxError: invalid syntax\r\r", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}}, "msg": "The following modules failed to execute: ansible.legacy.setup\n"} The tmp file has the following contents <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AuthorizationQueryParametersError</Code><Message>Error parsing the X-Amz-Credential parameter; the region 'us-east-1' is wrong; expecting 'eu-west-1'</Message><Region>eu-west-1</Region><RequestId>4VTAGR4C1V9ATBJT</RequestId><HostId>OahjGsFQHlr3ihxobH/yyH7Mzxq98mwjcb6+J3Y2EifDU7FykCe8b6QJTNodIG5WSquVeJF+Zsk=</HostId></Error> Steps to reproduce: run an ansible playbook using the following: aws_ec2 inventory aws_ssm connection type specify an s3 bucket (ansible_aws_ssm_bucket_name var) that is in a different location to the target node The presigned url generated includes the region the s3 bucket is in, so this region must be used for the session obtained in the _get_url function. Reviewed-by: Mark Chappell <None>
…n var (#1176) (#1290) [PR #1176/1be7da11 backport][stable-3] ssm connection: pull bucket region info rather than taking from region var This is a backport of PR #1176 as merged into main (1be7da1). SUMMARY Fix issue where syntax error is reported if using ssm connection and the target node is located in a different region to the s3 bucket. Fixes #1190, #637 ISSUE TYPE Bugfix Pull Request COMPONENT NAME aws_ssm connection plugin ADDITIONAL INFORMATION When using SSM for ansible connection and the target node is in a different region to the s3 bucket used, the playbook immediately errors with the following. (There are no issues when both target and s3 bucket are in the same region) fatal: [i-04444a7f03cc2bffd]: FAILED! => {"ansible_facts": {}, "changed": false, "failed_modules": {"ansible.legacy.setup": {"failed": true, "module_stderr": "", "module_stdout": " File \"/tmp/ansible/ansible-tmp-1653576081.8378458-29658-258097978113216/AnsiballZ_setup.py\", line 1\r\r\n <?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\r\n ^\r\r\nSyntaxError: invalid syntax\r\r", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}}, "msg": "The following modules failed to execute: ansible.legacy.setup\n"} The tmp file has the following contents <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AuthorizationQueryParametersError</Code><Message>Error parsing the X-Amz-Credential parameter; the region 'us-east-1' is wrong; expecting 'eu-west-1'</Message><Region>eu-west-1</Region><RequestId>4VTAGR4C1V9ATBJT</RequestId><HostId>OahjGsFQHlr3ihxobH/yyH7Mzxq98mwjcb6+J3Y2EifDU7FykCe8b6QJTNodIG5WSquVeJF+Zsk=</HostId></Error> Steps to reproduce: run an ansible playbook using the following: aws_ec2 inventory aws_ssm connection type specify an s3 bucket (ansible_aws_ssm_bucket_name var) that is in a different location to the target node The presigned url generated includes the region the s3 bucket is in, so this region must be used for the session obtained in the _get_url function. Reviewed-by: Mark Chappell <None>
|
I hit this issue yesterday when attempting to use the aws ssm connection with using a newly created bucket in us-east-2 yesterday. Can we reopen this issue? The referenced PR in the close action above (#1176) does not seem to resolve this particular issue. That PR was targeting a fix for pulling the region information for the S3 bucket used for SSM file transfers from the bucket metadata itself, but the pre-signed URLs generated for the S3 downloads are still pointing at the global S3 endpoint, and not the region specific one. Thus, attempting to use the ssm plugin with a newly created transfer bucket in a region like us-east-2 continues to return the presigned URLs targeting the global S3 endpoint, which results in the 307 redirect to the regional endpoint, which then causes the presigned URL to fail with a signature mismatch error as the url was signed for the global endpoint and not the regional endpoint. I believe that #1190 needs to be further addressed to resolve this issue. |
|
Hi @bodnarbm please see https://github.com/ansible-collections/community.aws/pull/743/files to use virtual addressing. There is a PR from @phene but that relies on a hardcoded region to be defined. |
|
@charles-paul-mox Thank you, but that pr looks to be closed unmerged and I would prefer to not patch the plugin separately (if I was I would probably add the s3 client endpoint url as a separate variable, that way I could also get it to work with other endpoints also (like fips endpoints)) I'm hoping that someone like @tremble could reopen this issue though. |
|
Yes, I cannot merge PRs due to company policies. The virtual addressing is the important part.
|
|
@charles-paul-mox My PR doesn't rely on a hard-coded region unless you are using a non-default partition like GovCloud. It uses the default global region just to query information about the S3 bucket's region, then uses the bucket's region from there on. |
|
Any real resolution to this problem? I'm using 5.1.0 release and there is still issue with AnsiballZ file: |
|
Any solution for this ? |
|
Cause:
to avoid this you can modify _get_boto_client() function client initializing as below to support addressing_style virtual for s3. client = session.client( |
…r auto. (#1633) Addressing Style S3 (ssm_connection) - choose between path, virtual or auto. SUMMARY Added the chance of setting up the addressing style for S3 URLs, this fixes #637 ISSUE TYPE Feature Pull Request COMPONENT NAME aws_ssm ADDITIONAL INFORMATION I came across this issue when I was trying to connect and execute tasks on a fresh AWS set-up (new S3 bucket + EC2), I'm using the latest (5.1.0) release and Ansible 5.10 (as per pip show). The issue itself is the same as #637 and the last comment is the fix, there's another PR addressing this issue #786 but is quiet since May Also AWS seems to be deprecating PATH addressing style: https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access Thanks to timburnet-systematica and james-masson for their initial work on this. Reviewed-by: Mark Chappell <None>
…r auto. (#1633) Addressing Style S3 (ssm_connection) - choose between path, virtual or auto. SUMMARY Added the chance of setting up the addressing style for S3 URLs, this fixes #637 ISSUE TYPE Feature Pull Request COMPONENT NAME aws_ssm ADDITIONAL INFORMATION I came across this issue when I was trying to connect and execute tasks on a fresh AWS set-up (new S3 bucket + EC2), I'm using the latest (5.1.0) release and Ansible 5.10 (as per pip show). The issue itself is the same as #637 and the last comment is the fix, there's another PR addressing this issue #786 but is quiet since May Also AWS seems to be deprecating PATH addressing style: https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access Thanks to timburnet-systematica and james-masson for their initial work on this. Reviewed-by: Mark Chappell <None> (cherry picked from commit 8503d80)
…r auto. (#1633) (#1669) [PR #1633/8503d80d backport][stable-5] Addressing Style S3 (ssm_connection) - choose between path, virtual or auto. This is a backport of PR #1633 as merged into main (8503d80). SUMMARY Added the chance of setting up the addressing style for S3 URLs, this fixes #637 ISSUE TYPE Feature Pull Request COMPONENT NAME aws_ssm ADDITIONAL INFORMATION I came across this issue when I was trying to connect and execute tasks on a fresh AWS set-up (new S3 bucket + EC2), I'm using the latest (5.1.0) release and Ansible 5.10 (as per pip show). The issue itself is the same as #637 and the last comment is the fix, there's another PR addressing this issue #786 but is quiet since May Also AWS seems to be deprecating PATH addressing style: https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access Thanks to timburnet-systematica and james-masson for their initial work on this. Reviewed-by: Mark Chappell <None>
Summary
When I try to execute a playbook against an Amazon Linux 2 instance in EC2 using the aws_ssm connection plugin and a recently created (less than an hour old) S3 bucket, it fails to correctly download
AnsiballZ_setup.py, resulting in a python syntax error" File \"/home/ssm-user/.ansible/tmp/ansible-tmp-1626190404.700778-20074-247496938615569/AnsiballZ_setup.py\", line 1\r\r\n <?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\r\n ^\r\r\nSyntaxError: invalid syntax\r\r",.The curl is writing out the S3 XML error response to file, due to S3 returning a HTTP 307 redirect which the curl does not follow.
This HTTP 307 from S3 is expected, as per this AWS documentation, because the bucket is too new for the global S3 DNS to have propagated out yet, so a regional endpoint has to be used.
This overall seems similar to this issue, but is still happening for me when using the
mainbranch of this repository where the fix has been applied.I believe the underlying problem is that when the signed url is generated in the function
_file_transport_command, it is a global URL rather than a regional URL:For example, the URL below does not work and returns a 307;
https://test-bucket-garethsaxby-20210713-153159.s3.amazonaws.com/i-089c1ec0c85524f5d//home/ssm-user/.ansible/tmp/ansible-tmp-1626189520.659307-19800-45563405192193/AnsiballZ_setup.py?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA5QGXVMSCMPOQVZH3%2F20210713%2Feu-west-2%2Fs3%2Faws4_request&X-Amz-Date=20210713T151841Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=18d520a539227540bef2ba06a6000dd6569c868aeb4cc6ae042fb895e5e2f880
Whilst the URL below, redirected by the 307, -does- work;
https://test-bucket-garethsaxby-20210713-153159.s3.eu-west-2.amazonaws.com/i-089c1ec0c85524f5d//home/ssm-user/.ansible/tmp/ansible-tmp-1626189520.659307-19800-45563405192193/AnsiballZ_setup.py?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA5QGXVMSCMPOQVZH3%2F20210713%2Feu-west-2%2Fs3%2Faws4_request&X-Amz-Date=20210713T151841Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=18d520a539227540bef2ba06a6000dd6569c868aeb4cc6ae042fb895e5e2f880
If I force the plugin to use a regional endpoint for S3, and use a region when creating the client, as per my branch, it does work, albeit I'm not really sure -how- best to implement this to properly put a Pull Request together to fix the problem, given my branch feels like a really ugly hack.
Issue Type
Bug Report
Component Name
plugins/connection/aws_ssm
Ansible Version
Collection Versions
AWS SDK versions
Configuration
OS / Environment
Client: macOS Catalina 10.15.7, Ansible installed via Brew
Remote: Amazon Linux 2, eu-west-2, ami-03ac5a9b225e99b02, amzn2-ami-hvm-2.0.20210701.0-x86_64-gp2
Steps to Reproduce
requirements.yml:
ansible.cfg:
inventory.aws_ec2.yml:
playbook.yml
Expected Results
I'm expecting the curl against the S3 signed URL on the remote host to pull down
AnsiballZ_setup.pycorrectly and continue running the playbook, returning the ping successfully.Actual Results
When I curl the signed URL from the remote instance using SSM Session Manager, I get the following response, showing that a 307 is being returned and I'm being redirected to the regional endpoint:
Code of Conduct
The text was updated successfully, but these errors were encountered: