"unknown" error message accessing S3 object

[asieira]

I have run into a strange problem with using the S3 functionality. To understand what's happening, I have a bucket where allow was granted for s3:* for a given user. I have configured that user's credentials in ~/.aws/config as a profile called prof.

I am able to perform list operations (s3 aws ls), upload files (s3 aws cp), etc. However, I am not able to download files for some reason. I was able to map that to an

 | => aws --profile prof --region us-west-2 s3api get-object --bucket my-bucket --key 'dir1/dir2/file.ext' file.ext --debug
2014-01-28 12:53:43,947 - awscli.clidriver - DEBUG - CLI version: aws-cli/1.2.11 Python/2.7.6 Darwin/13.0.0, botocore version: 0.31.0
2014-01-28 12:53:43,947 - botocore.service - DEBUG - Creating service object for: s3
2014-01-28 12:53:43,947 - botocore.base - DEBUG - Attempting to load: aws/s3
2014-01-28 12:53:43,971 - botocore.base - DEBUG - Found data file: /usr/local/lib/python2.7/site-packages/botocore/data/aws/s3.json
2014-01-28 12:53:43,971 - botocore.hooks - DEBUG - Event service-created: calling handler <function register_retries_for_service at 0x10e88e938>
2014-01-28 12:53:43,971 - botocore.handlers - DEBUG - Registering retry handlers for service: Service(s3)
2014-01-28 12:53:43,971 - botocore.service - DEBUG - Creating operation objects for: Service(s3)
2014-01-28 12:53:43,977 - botocore.operation - DEBUG - Creating parameter objects for: Operation:GetObject
2014-01-28 12:53:43,978 - awscli.clidriver - DEBUG - OrderedDict([(u'bucket', <awscli.arguments.CLIArgument object at 0x10ec56250>), (u'if-match', <awscli.arguments.CLIArgument object at 0x10ec566d0>), (u'if-modified-since', <awscli.arguments.CLIArgument object at 0x10ec56710>), (u'if-none-match', <awscli.arguments.CLIArgument object at 0x10ec56750>), (u'if-unmodified-since', <awscli.arguments.CLIArgument object at 0x10ec56790>), (u'key', <awscli.arguments.CLIArgument object at 0x10ec567d0>), (u'range', <awscli.arguments.CLIArgument object at 0x10ec56810>), (u'response-cache-control', <awscli.arguments.CLIArgument object at 0x10ec56850>), (u'response-content-disposition', <awscli.arguments.CLIArgument object at 0x10ec568d0>), (u'response-content-encoding', <awscli.arguments.CLIArgument object at 0x10ec56910>), (u'response-content-language', <awscli.arguments.CLIArgument object at 0x10ec56950>), (u'response-content-type', <awscli.arguments.CLIArgument object at 0x10ec56990>), (u'response-expires', <awscli.arguments.CLIArgument object at 0x10ec569d0>), (u'version-id', <awscli.arguments.CLIArgument object at 0x10ec56a10>)])
2014-01-28 12:53:43,978 - botocore.hooks - DEBUG - Event building-argument-table.s3api.get-object: calling handler <function add_streaming_output_arg at 0x10ead22a8>
2014-01-28 12:53:43,978 - botocore.hooks - DEBUG - Event building-argument-table.s3api.get-object: calling handler <function unify_paging_params at 0x10ead2aa0>
2014-01-28 12:53:43,980 - botocore.hooks - DEBUG - Event process-cli-arg.s3.get-object: calling handler <awscli.argprocess.ParamShorthand object at 0x10eb584d0>
2014-01-28 12:53:43,980 - awscli.arguments - DEBUG - Unpacked value of "my-bucket" for parameter "bucket": my-bucket
2014-01-28 12:53:43,980 - botocore.hooks - DEBUG - Event process-cli-arg.s3.get-object: calling handler <awscli.argprocess.ParamShorthand object at 0x10eb584d0>
2014-01-28 12:53:43,980 - awscli.arguments - DEBUG - Unpacked value of "dir1/dir2/file.ext" for parameter "key": dir1/dir2/file.ext
2014-01-28 12:53:43,980 - botocore.credentials - INFO - Found credentials in config file.
2014-01-28 12:53:43,981 - botocore.hooks - DEBUG - Event creating-endpoint.s3: calling handler <function maybe_switch_to_s3sigv4 at 0x10e88ea28>
2014-01-28 12:53:44,000 - botocore.operation - DEBUG - Operation:GetObject called with kwargs: {u'bucket': 'my-bucket', u'key': 'dir1/dir2/file.ext'}
2014-01-28 12:53:44,000 - botocore.endpoint - DEBUG - Making request for Operation:GetObject (verify_ssl=True) with params: {'headers': {}, 'uri_params': {u'Bucket': 'my-bucket', u'Key': 'dir1/dir2/file.ext'}, 'payload': <botocore.payload.XMLPayload object at 0x10ec5ae10>}
2014-01-28 12:53:44,000 - botocore.endpoint - DEBUG - Building URI for rest endpoint.
2014-01-28 12:53:44,000 - botocore.endpoint - DEBUG - Templated URI path: /{Bucket}/{Key}
2014-01-28 12:53:44,000 - botocore.endpoint - DEBUG - Templated URI query_params: versionId={VersionId}&response-content-type={ResponseContentType}&response-content-language={ResponseContentLanguage}&response-expires={ResponseExpires}&response-cache-control={ResponseCacheControl}&response-content-disposition={ResponseContentDisposition}&response-content-encoding={ResponseContentEncoding}
2014-01-28 12:53:44,000 - botocore.endpoint - DEBUG - Rendered path: /my-bucket/dir1/dir2/file.ext
2014-01-28 12:53:44,001 - botocore.endpoint - DEBUG - Rendered query_params: 
2014-01-28 12:53:44,001 - botocore.hooks - DEBUG - Event before-auth.s3: calling handler <function fix_s3_host at 0x10e88e848>
2014-01-28 12:53:44,001 - botocore.handlers - DEBUG - Checking for DNS compatible bucket for: https://s3-us-west-2.amazonaws.com/my-bucket/dir1/dir2/file.ext
2014-01-28 12:53:44,001 - botocore.handlers - DEBUG - URI updated to: https://my-bucket.s3.amazonaws.com/dir1/dir2/file.ext
2014-01-28 12:53:44,001 - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2014-01-28 12:53:44,001 - botocore.auth - DEBUG - HTTP request method: GET
2014-01-28 12:53:44,001 - botocore.auth - DEBUG - StringToSign:
GET


Tue, 28 Jan 2014 20:53:44 GMT
/my-bucket/dir1/dir2/file.ext
2014-01-28 12:53:44,013 - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [GET]>
2014-01-28 12:53:44,014 - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): my-bucket.s3.amazonaws.com
2014-01-28 12:53:44,795 - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /dir1/dir2/file.ext HTTP/1.1" 403 None
2014-01-28 12:53:44,796 - botocore.hooks - DEBUG - Event needs-retry.s3.GetObject: calling handler <botocore.retryhandler.RetryHandler object at 0x10eb68a10>
2014-01-28 12:53:44,796 - botocore.retryhandler - DEBUG - No retry needed.
2014-01-28 12:53:44,796 - botocore.hooks - DEBUG - Event after-call.s3.GetObject: calling handler <awscli.errorhandler.ErrorHandler object at 0x10eb58650>
2014-01-28 12:53:44,796 - awscli.errorhandler - DEBUG - HTTP Response Code: 403
2014-01-28 12:53:44,796 - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/awscli/clidriver.py", line 188, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/usr/local/lib/python2.7/site-packages/awscli/clidriver.py", line 331, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
  File "/usr/local/lib/python2.7/site-packages/awscli/clidriver.py", line 439, in __call__
    self._operation_object, call_parameters, parsed_globals)
  File "/usr/local/lib/python2.7/site-packages/awscli/clidriver.py", line 530, in invoke
    **parameters)
  File "/usr/local/lib/python2.7/site-packages/botocore/operation.py", line 82, in call
    parsed=response[1])
  File "/usr/local/lib/python2.7/site-packages/botocore/session.py", line 613, in emit
    return self._events.emit(event_name, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/botocore/hooks.py", line 158, in emit
    response = handler(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/awscli/errorhandler.py", line 75, in __call__
    http_status_code=http_response.status_code)
ClientError: A client error (Unknown) occurred when calling the GetObject operation: Unknown
2014-01-28 12:53:44,798 - awscli.clidriver - DEBUG - Exiting with rc 255
A client error (Unknown) occurred when calling the GetObject operation: Unknown

[asieira]

Just wanted to let you know that I deleted the files and re-uploaded them, and then everything worked. Very weird.

Since S3 doesn't guarantee 100% consistency of data, maybe I got 'lucky' and had a corrupted one? Anyway, might be worth investigating if it is possible to detect that situation and provide a clearer error message on awscli.

[asieira]

Turns out that this is related to #663.

The bucket was created in AWS account A.

The instance doing the upload runs in AWS account B, and uses an IAM role also that also belongs to AWS account B.

The bucket has a bucket policy granting s3:* to the aforementioned IAM role.

I was getting an error reading this file using credentials from an IAM user from AWS account A.

The correct way to ensure all privileges were correct (AFAIK) would be to use the bucket-owner-full-control canned ACL, which gives full rights to both the bucket owner AWS account (A) and object owner AWS account (B).

Still, the error message could have been better than "unknown". But knowing the case and being able to replicate it might help. :)

[jamesls]

Should be fixed via boto/botocore#254