Uploading files to S3 from EC2 instances fails on some instance types

[benbc]

We have a very strange problem. We are uploading large (10M) files from an EC2 instance to an S3 bucket in the same region. We use aws s3 cp for the upload and instance roles for authentication.

If we use an m1.small EC2 instance the upload works fine. But if we increase the instance size (we have tried m1.large and m3.large) the upload fails.

Here is the error we get:

upload failed: <file> to <s3-address>
HTTPSConnectionPool(host='<bucket>.s3.amazonaws.com', port=443): Max retries exceeded with url: <s2-key>?partNumber=2&uploadId=Hw6FZhXeH03ZzzCtxu2XcsspmpN2lx83GxWGzrJ96I81XlajjjNCxsSj0vs0BIbf_xJmzfdZwZY_UzUL3NCT57rpqEwIuLD1qryr7KKjlR6rA0YIW_ltopZQkoo32i0qTPUAlBc55WRAYxjwJ8Iw1A-- (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer)

This is completely reproducible -- we have never had such a upload succeed after tens of attempts. We have never seen the problem on an m1.small instance in hundreds of attempts.

We ran into this problem with 10M files. We have found that it is reproducible down to about 1M. Much less than this and it works fine every time.

Any ideas about what is going on would be much appreciated.

[jamesls]

What version of the CLI are you using? This looks like #594, which was fixed in 1.2.11

[jamesls]

Actually, that issue was for downloads, this one is for uploads so it's likely a different issue. I'll need to look into this further. If you could provide a --debug log of this, it would help in troubleshooting.

Also approximately how many 10M files are you uploading at once?

[joshma]

I'm also having issues uploading to s3, but from my laptop (OSX mavericks). A 424KB file failed,
and a 344KB file worked. I'm uploading one file at a time, with the following versions:

awscli==1.2.13
boto==2.24.0
botocore==0.33.0

The error is the same as the original post:

upload failed: ./test to s3://<bucket>/deploys/test HTTPSConnectionPool(host='bnch.s3.amazonaws.com', port=443): Max retries exceeded with url: /deploys/test (Caused by <class 'socket.error'>: [Errno 32] Broken pipe)

I ran it with --debug, and here's the part that keeps on retrying:

2014-02-06 21:20:17,138 - botocore.retryhandler - DEBUG - Retry needed, action of: 3.28890874908
2014-02-06 21:20:17,138 - botocore.endpoint - DEBUG - Response received to retry, sleeping for 3.28890874908 seconds
2014-02-06 21:20:20,428 - botocore.awsrequest - DEBUG - Rewinding stream: <open file u'<redacted>/test', mode 'rb' at 0x101dc6c90>
2014-02-06 21:20:20,428 - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [PUT]>
2014-02-06 21:20:20,429 - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (4): <bucket>.s3.amazonaws.com
2014-02-06 21:20:21,693 - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x101e0d550>
2014-02-06 21:20:21,693 - botocore.retryhandler - DEBUG - retry needed, retryable exception caught: HTTPSConnectionPool(host='<bucket>.s3.amazonaws.com', port=443): Max retries exceeded with url: /deploys/test (Caused by <class 'socket.error'>: [Errno 32] Broken pipe)
Traceback (most recent call last):
  File "<redacted>/lib/python2.7/site-packages/botocore/retryhandler.py", line 262, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "<redacted>/lib/python2.7/site-packages/botocore/retryhandler.py", line 310, in __call__
    caught_exception)
  File "<redacted>/lib/python2.7/site-packages/botocore/retryhandler.py", line 219, in __call__
    return self._check_caught_exception(attempt_number, caught_exception)
  File "<redacted>/lib/python2.7/site-packages/botocore/retryhandler.py", line 352, in _check_caught_exception
    raise caught_exception
ConnectionError: HTTPSConnectionPool(host='<bucket>.s3.amazonaws.com', port=443): Max retries exceeded with url: /deploys/test (Caused by <class 'socket.error'>: [Errno 32] Broken pipe)
2014-02-06 21:20:21,693 - botocore.retryhandler - DEBUG - Retry needed, action of: 2.598356941
2014-02-06 21:20:21,693 - botocore.endpoint - DEBUG - Response received to retry, sleeping for 2.598356941 seconds
2014-02-06 21:20:24,293 - botocore.awsrequest - DEBUG - Rewinding stream: <open file u'<redacted>/test', mode 'rb' at 0x101dc6c90>
2014-02-06 21:20:24,293 - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [PUT]>
2014-02-06 21:20:24,294 - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (5): <bucket>.s3.amazonaws.com
2014-02-06 21:20:25,888 - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x101e0d550>
2014-02-06 21:20:25,888 - awscli.customizations.s3.tasks - DEBUG - <redacted>/test upload failure: HTTPSConnectionPool(host='<bucket>.s3.amazonaws.com', port=443): Max retries exceeded with url: /deploys/test (Caused by <class 'socket.error'>: [Errno 32] Broken pipe)
2014-02-06 21:20:25,888 - botocore.operation - DEBUG - Operation:PutObject called with kwargs: {'body': <open file u'<redacted>/test', mode 'rb' at 0x101e61d20>, 'bucket': u'<bucket>', 'key': u'deploys/test'}
2014-02-06 21:20:25,889 - botocore.endpoint - DEBUG - Making request for Operation:PutObject (verify_ssl=True) with params: {'headers': {}, 'uri_params': {u'Bucket': u'<bucket>', u'Key': u'deploys/test'}, 'payload': <botocore.payload.Payload object at 0x101fc3ed0>}
2014-02-06 21:20:25,889 - botocore.endpoint - DEBUG - Building URI for rest endpoint.
2014-02-06 21:20:25,889 - botocore.endpoint - DEBUG - Templated URI path: /{Bucket}/{Key}
2014-02-06 21:20:25,889 - botocore.endpoint - DEBUG - Templated URI query_params:
2014-02-06 21:20:25,889 - botocore.endpoint - DEBUG - Rendered path: /<bucket>/deploys/test
2014-02-06 21:20:25,889 - botocore.endpoint - DEBUG - Rendered query_params:
2014-02-06 21:20:25,890 - botocore.hooks - DEBUG - Event before-auth.s3: calling handler <function fix_s3_host at 0x101b24410>
2014-02-06 21:20:25,890 - botocore.handlers - DEBUG - Checking for DNS compatible bucket for: https://s3-us-west-2.amazonaws.com/<bucket>/deploys/test
2014-02-06 21:20:25,890 - botocore.handlers - DEBUG - URI updated to: https://<bucket>.s3.amazonaws.com/deploys/test
2014-02-06 21:20:25,890 - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2014-02-06 21:20:25,890 - botocore.auth - DEBUG - HTTP request method: PUT
2014-02-06 21:20:25,890 - botocore.auth - DEBUG - StringToSign:
PUT


Fri, 07 Feb 2014 05:20:25 GMT
/<bucket>/deploys/test
2014-02-06 21:20:25,891 - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [PUT]>
2014-02-06 21:20:25,891 - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (6): <bucket>.s3.amazonaws.com
2014-02-06 21:20:27,373 - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x101e0d550>
2014-02-06 21:20:27,374 - botocore.retryhandler - DEBUG - retry needed, retryable exception caught: HTTPSConnectionPool(host='<bucket>.s3.amazonaws.com', port=443): Max retries exceeded with url: /deploys/test (Caused by <class 'socket.error'>: [Errno 32] Broken pipe)
Traceback (most recent call last):
  File "<redacted>/lib/python2.7/site-packages/botocore/retryhandler.py", line 262, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "<redacted>/lib/python2.7/site-packages/botocore/retryhandler.py", line 310, in __call__
    caught_exception)
  File "<redacted>/lib/python2.7/site-packages/botocore/retryhandler.py", line 219, in __call__
    return self._check_caught_exception(attempt_number, caught_exception)
  File "<redacted>/lib/python2.7/site-packages/botocore/retryhandler.py", line 352, in _check_caught_exception
    raise caught_exception
ConnectionError: HTTPSConnectionPool(host='<bucket>.s3.amazonaws.com', port=443): Max retries exceeded with url: /deploys/test (Caused by <class 'socket.error'>: [Errno 32] Broken pipe)
2014-02-06 21:20:27,374 - botocore.retryhandler - DEBUG - Retry needed, action of: 0.627294998584
2014-02-06 21:20:27,374 - botocore.endpoint - DEBUG - Response received to retry, sleeping for 0.627294998584 seconds

[benbc]

We are using version 1.2.13. We have about 200 of these files to upload, but we are doing it in series. The problem always happens on the first one.

Here is the debug log of the failure that you asked for: https://gist.github.com/benbc/8861033.

[zindello]

Is there any update on this issue. I'm seeing Exactly the same issue trying to upload to a newly created bucket.

[jamesls]

Yes I believe this should be fixed now in version 1.3.0. The root cause I believe is the same as #544.

Can you please try with the latest CLI version. If you're still seeing the issue, please attach a --debug logs and I'll repoen the issue.

[hola8352]

i'm still having this issue upload to s3, neither from my laptop ,nor from debian server .The file is about 100MB. I'm uploading one file at a time, with the following versions:

   aws-cli/1.3.1 Python/2.7.3 Linux/3.2.0-4-amd64
   aws-cli/1.3.1 Python/2.7.5 Darwin/13.0.0

Here is the issue I get:

Max retries exceeded with url: <myfile>?partNumber=13&uploadId=bR1a29GdR1Qb0CW4fSDv9Vvi4PKJLF3eizWErF1SGvuMxUAh4vmHLmVQ7XjVSsFGmQi0N1U8KX5dR1uukEAMaAS8JxoQA89nJN2FFmQu2MRzIlrIzeMr4re5Zwvn3Hvv.n9IKm.OXOqy4NN87yohIA-- (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer)

Here is the debug log:

Wed, 19 Mar 2014 03:28:03 GMT
/<mybucket>/<myfile>?partNumber=9&uploadId=psq_ysXJSW9eplwLTW97ueGw1BMzG2qEXVoP9XiQsm06RP0_N_J_qPS1vZf4OOj0W33Bwdey2w4KNsbEltH_GIWO3jOQbcP64MEfTMd.OSUVsmZKsMIWCxZoqdbmspb1bD0YzAbG92F9R8pQQrXdkA--
2014-03-19 11:28:03,343 - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [PUT]>
partNumber=8&uploadId=psq_ysXJSW9eplwLTW97ueGw1BMzG2qEXVoP9XiQsm06RP0_N_J_qPS1vZf4OOj0W33Bwdey2w4KNsbEltH_GIWO3jOQbcP64MEfTMd.OSUVsmZKsMIWCxZoqdbmspb1bD0YzAbG92F9R8pQQrXdkA-- (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 262, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 310, in __call__
    caught_exception)
  File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 219, in __call__
    return self._check_caught_exception(attempt_number, caught_exception)
  File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 352, in _check_caught_exception
    raise caught_exception
ConnectionError: HTTPSConnectionPool(host='<mybucket>.s3.amazonaws.com', port=443): Max retries exceeded with url: /<myfile>?partNumber=8&uploadId=psq_ysXJSW9eplwLTW97ueGw1BMzG2qEXVoP9XiQsm06RP0_N_J_qPS1vZf4OOj0W33Bwdey2w4KNsbEltH_GIWO3jOQbcP64MEfTMd.OSUVsmZKsMIWCxZoqdbmspb1bD0YzAbG92F9R8pQQrXdkA-- (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer)
2014-03-19 11:28:05,374 - botocore.retryhandler - DEBUG - Retry needed, action of: 0.41042383996
2014-03-19 11:28:05,374 - botocore.endpoint - DEBUG - Response received to retry, sleeping for 0.41042383996 seconds
2014-03-19 11:28:05,579 - botocore.hooks - DEBUG - Event needs-retry.s3.UploadPart: calling handler <botocore.retryhandler.RetryHandler object at 0x308bfd0>
2014-03-19 11:28:05,580 - botocore.retryhandler - DEBUG - retry needed, retryable exception caught: HTTPSConnectionPool(host=<mybucket>.s3.amazonaws.com', port=443): Max retries exceeded with url: /<myfile>?partNumber=1&uploadId=psq_ysXJSW9eplwLTW97ueGw1BMzG2qEXVoP9XiQsm06RP0_N_J_qPS1vZf4OOj0W33Bwdey2w4KNsbEltH_GIWO3jOQbcP64MEfTMd.OSUVsmZKsMIWCxZoqdbmspb1bD0YzAbG92F9R8pQQrXdkA-- (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 262, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 310, in __call__
    caught_exception)
  File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 219, in __call__
    return self._check_caught_exception(attempt_number, caught_exception)
  File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 352, in _check_caught_exception
    raise caught_exception
ConnectionError: HTTPSConnectionPool(host='<mybucket>.s3.amazonaws.com', port=443): Max retries exceeded with url: /<myfile>?partNumber=1&uploadId=psq_ysXJSW9eplwLTW97ueGw1BMzG2qEXVoP9XiQsm06RP0_N_J_qPS1vZf4OOj0W33Bwdey2w4KNsbEltH_GIWO3jOQbcP64MEfTMd.OSUVsmZKsMIWCxZoqdbmspb1bD0YzAbG92F9R8pQQrXdkA-- (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer)

[here]

Still seeing this issue using aws cp and aws sync on a new bucket. Smaller files went through no problem, larger files reliably fail as described here and #544

aws --version
aws-cli/1.3.3 Python/2.7.5+ Linux/3.8.0-35-generic

>>> print botocore.__version__
>>> 0.37.0

upload failed: ./file.tar.gz to s3://mybucket/file.tar.gz
HTTPSConnectionPool(host='mybucket.s3.amazonaws.com', port=443): Max retries exceeded with url: /file.tar.gz?partNumber=... (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer)

curl https://mybucket.s3.amazonaws.com
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>TemporaryRedirect</Code><Message>Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests.</Message><RequestId>...</RequestId><Bucket>mybucket</Bucket><HostId>...</HostId><Endpoint>mybucket.s3-us-west-1.amazonaws.com</Endpoint></Error>

[here]

As others have reported, this is now working fine after the TemporaryRedirect code clears --

$ curl https://mybucket.s3.amazonaws.com
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>...</RequestId><HostId>...</HostId></Error>%

[corruptmem]

I'm still encountering this problem with version 1.3.4 on a new Ireland bucket.

ubuntu@ip-172-31-40-164:~$ aws --version
aws-cli/1.3.4 Python/2.7.5+ Linux/3.11.0-15-generic

Debug log: https://gist.github.com/anonymous/9814978

[jeekajoo]

Same bug for me when using 'aws s3 sync'
$ aws --version
aws-cli/1.3.4 Python/2.7.3 Linux/3.2.0-57-virtual

My bucket is in ireland.

debug log: https://paste.buntux.org/?c511ef0a27305636#e1/B5RiyBuZq60LHmLNpZt59wz2zDyY2KVdjCM+mO7E=

edit: I tested on a bucket in "US Standard" region --> no problem
edit2: new url for debug log https://paste.jeekajoo.eu/?c511ef0a27305636#e1/B5RiyBuZq60LHmLNpZt59wz2zDyY2KVdjCM+mO7E=

[jamesls]

I'm reopening this issue. After debugging this I can confirm what others have said. This problem exists when trying to upload a large file to a newly created bucket that's not in the classic region.

From what I can tell the CLI is properly retrying requests and following 307 redirects. The problem, however, is that the CLI sends the entire request and then waits for a response. However, S3 will immediately send the 307 response before we've finished sending the body. Eventually it will just close the connection, and if we're still in the process of streaming the body, we will not see the response. Instead we get the ConnectionError as shown in the various debug logs above.

The normal way to address this would be to use the expect 100 continue header. However, the HTTP client we use (requests) does not support this. There might be a way to work around this, but I'll need to do some digging into the requests library to see the best way to fix this issue.

I'll update this issue when I know more.

[pex]

Any news?

[mjpan]

my tests show that even when the boto library is failing, the AWS CLI works fine. what is the AWS CLI doing differently?

[jamesls]

I don't think that there's any ways to workaround with in the requests library. I think our best option (aside from writing our own HTTP client that supports expect 100 continue) is some specific workaround in the S3 code.

[pwaller]

I've just hit this too. My file isn't large, just a few megabytes.

[danielsamuels]

Getting this with a 323KB CSS file.

[jamesls]

Working on a fix for this, will update when I have more info.

[swkimme]

Getting same issue when uploading large file (3GB) into S3. Uploading small file is just fine.

upload failed: ../../tmp/anonymousAction.log.2014-05-12.tokyo1b-http-4-1.20140513T003001 to s3://between-data-analysis-temp/anonymousAction.log.2014-05-12.tokyo1b-http-4-1.20140513T003001
HTTPSConnectionPool(host='between-data-analysis-temp.s3.amazonaws.com', port=443): Max retries exceeded with url: /anonymousAction.log.2014-05-12.tokyo1b-http-4-1.20140513T003001?partNumber=6&uploadId=Wdj8309FlXToB8mXw6JHnz8Xf8f4ADIC2SiaM3a6foSDJXH9t7xLk2uFK_EgTtjMn.OyCWdTJjP9al3rgISEltpMwj9j77Irfjc5wqssG2zgnUKhXM2gEUU5QfAuH0kuJNg8deDipYvwBkE0r7k3Qw-- (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer)
... 

My aws version;
aws-cli/1.3.13 Python/2.7.3 Linux/3.2.0-61-virtual

[sd2k]

I'm getting a similar issue intermittently (about 1 file in every 20, files are around 500MB) using the requests library directly as we're working through our specific storage API. This is uploading from an EC2 instance.

  File "/data/opi/upload.py", line 174, in upload
    response = requests.put(url, data=data, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests-2.1.0-py2.7.egg/requests /api.py", line 99, in put
    return request('put', url, data=data, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests-2.1.0-py2.7.egg/requests /api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests-2.1.0-py2.7.egg/requests /sessions.py", line 382, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests-2.1.0-py2.7.egg/requests /sessions.py", line 485, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests-2.1.0-py2.7.egg/requests /adapters.py", line 372, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='our-bucket.s3.amazonaws.com', port=443): Max retries exceeded with url: /folder/subfolder/filename?Signature=XXX&Expires=1401262800&AWSAccessKeyId=YYY -amz-acl=private (Caused by <class 'socket.error'>: [Errno 104] Connection reset  by peer)

[jamesls]

Just an update, still working on this. The most viable approach so far has been to subclass HTTPConnection and try to implement the expect 100-continue logic in our subclass. We'd then plumb this into urllib3's connectionpools, which will be picked up by requests (which is the http library we use).

So I believe I have a potential fix, but it needs more testing. Will update when I have more info.

[pwaller]

@jamesls as soon as you have a public branch to test, I'll be using it :)

[jan-carreras]

Hi there,

since its not explicitly said in the thread, you can still use the "aws" command to upload files to another region by specifying the --region param.

• This one FAILS
  aws s3 cp ./really_big_file s3://MY-BUCKET/
• This one WORKS
  aws --region eu-west-1 s3 cp ./really_big_file s3://MY-BUCKET/

[pwaller]

@inedit00 Thanks! I had no idea that this was related to the problem.

[jamesls]

The fix for this is here: boto/botocore#303

But I'd like to do more testing on this. I also want to see what sending the expect 100-continue header does to perf.

[nylund]

We are getting this error also when copying a large file (~3GB) from local to S3 (Ireland). Using aws-cli/1.3.17 Python/2.6.6 Linux/2.6.32-71.29.1.el6.x86_64 and botocore 0.51.0.

[jamesls]

Just a quick update, the fix for this was in boto/botocore#303, however we caught a regression that was only in python2.6 so we had to revert the PR.

I think it's still possible to fix this, but it will be slightly more invasive due to internal changes in httplib from python2.6 to >python2.6. I'll update this when I have more info.

[jamesls]

I was able to fix the python2.6 issue in the previous PR to fix this issue. All of the tests are passing with this change, and I can confirm that this fixes the issue of not being able to upload large files to newly created non-standard S3 buckets. Please let me know if you're still seeing issues, and thanks for your patience while we resolved this issue.

[benbc]

Thanks for the fix @jamesls. Which awscli release is/will this fix be in?

[jamesls]

This is available in the latest version of the CLI (v1.3.18).

Also, if anyone is needing to debug this in the future, you'll see debug logs related to the expect 100 continue header:

2014-06-23 10:40:32,495 - Thread-4 - botocore.hooks - DEBUG - Event before-call.s3.UploadPart: calling handler <function add_expect_header at 0x10d3469b0>
2014-06-23 10:40:32,495 - Thread-4 - botocore.handlers - DEBUG - Adding expect 100 continue header to request.
2014-06-23 10:40:32,500 - Thread-4 - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [PUT]>
2014-06-23 10:40:32,502 - Thread-4 - botocore.awsrequest - DEBUG - Waiting for 100 Continue response.
2014-06-23 10:40:32,506 - Thread-4 - botocore.awsrequest - DEBUG - Received a non 100 Continue response from the server, NOT sending request body.
2014-06-23 10:40:32,512 - Thread-4 - botocore.awsrequest - DEBUG - Redirect received, rewinding stream: <awscli.customizations.s3.utils.ReadFileChunk object at 0x10d7f1450>
2014-06-23 10:40:32,512 - Thread-4 - botocore.awsrequest - DEBUG - Rewinding stream: <awscli.customizations.s3.utils.ReadFileChunk object at 0x10d7f1450>
2014-06-23 10:40:32,513 - Thread-4 - botocore.awsrequest - DEBUG - Waiting for 100 Continue response.
2014-06-23 10:40:32,533 - Thread-4 - botocore.awsrequest - DEBUG - 100 Continue response seen, now sending request body.
...
2014-06-23 10:40:34,530 - Thread-4 - awscli.errorhandler - DEBUG - HTTP Response Code: 200

[ErikEvenson]

I believe this is still a problem per #823.

[kevin-canadian]

That looks like a different problem. The errno is different, in particular.

[KeithTt]

got the same issue.

upload failed: images1/1103/142_b2d35e8005321b582ce891ca835df65d_75.jpg to s3://opgirl-v2/images/images1/1103/142_b2d35e8005321b582ce891ca835df65d_75.jpg A client error (RequestTimeout) occurred when calling the PutObject operation: Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.
Cleaning up. Please wait...
upload failed: images1/1103/142_d0754d4c8708dc2d7c68cc27da2ad2fa_100.jpg to s3://opgirl-v2/images/images1/1103/142_d0754d4c8708dc2d7c68cc27da2ad2fa_100.jpg The write operation timed out
upload failed: images1/1103/1430_0a6a493f08b693789cba32fbbc6ade83_75.jpg to s3://opgirl-v2/images/images1/1103/1430_0a6a493f08b693789cba32fbbc6ade83_75.jpg ('Connection aborted.', error(104, 'Connection reset by peer'))

# aws --version
aws-cli/1.9.15 Python/2.6.6 Linux/3.2.0-0.bpo.1-amd64 botocore/1.3.15

Reaching the performance limit of AWS s3?
https://stackoverflow.com/questions/23397460/error-handling-boto-error-104-connection-reset-by-peer

[jaakkovarjo]

I'm experiencing the same issue. This thread is rather old. Is this issue still being investigated?

('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

# aws --version aws-cli/1.11.13 Python/3.5.2 Linux/4.4.0-1055-aws botocore/1.4.70

[adilbaig]

Faced the same issue

> aws s3 cp /opt/tradetracker/dataStream/tracker-hits-2018-04-30_11:00:00.log s3://tt-feeder-dumps
upload failed: ../dataStream/tracker-hits-2018-04-30_11:00:00.log to s3://tt-feeder-dumps/tracker-hits-2018-04-30_11:00:00.log ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

> ls -hal /opt/tradetracker/dataStream/tracker-hits-2018-04-30_11:00:00.log
-rw-rw-rw- 1 www-data www-data 13M Apr 30 13:56 /opt/tradetracker/dataStream/tracker-hits-2018-04-30_11:00:00.log

> aws --version
aws-cli/1.11.13 Python/3.5.2 Linux/4.4.0-1049-aws botocore/1.4.70

[bilyum]

Ditto. +1
Filesize: 1G
aws-cli/1.15.12 Python/3.4.3 Linux/4.4.0-71-generic botocore/1.10.12

---

For my setup this was resolved by setting:
s3 =
max_concurrent_requests = 5
max_bandwidth = 210KB/s

in the .aws/config file for the profile I was using. See here: https://docs.aws.amazon.com/cli/latest/topic/s3-config.html#

[CSRaghunandan]

I'm having the same issue as well, when I tried to upload a file to s3 bucket using boto3 python package.

It used to work fine until about 4 days ago, and since then having problems with uploading.

[uldall]

I am having the same issue with a newly created bucket in the eu-west-1 region.

Command used:
aws s3 sync . s3://<bucket_name> --profile test --region eu-west-1

$ aws --version
aws-cli/1.11.13 Python/3.5.2 Linux/4.13.0-45-generic botocore/1.4.70

[CSRaghunandan]

Is this problem related to the network I'm connected to? The upload was failing when I had connected to a 4G dongle, whereas when I was connected to my wifi router, everything was fine.

But, the 4G dongle was giving me 10Mbps speed. So, not sure where the problem is

[chrisairpos]

@uldall If you haven't already solved this, for me this was fixed by installing the latest version of the aws-cli - the latest version on apt-get is outdated and so you must install using pip. Here's the instructrions:
https://www.slashroot.in/how-to-install-and-configure-aws-cli-on-ubuntu-16-04

[kenny1323]

On ubuntu 18
Install python 3
sudo apt-get install python3-pip
pip3 install awscli --upgrade --user
aws s3 sync ./ s3://mybucket01

https://linuxhint.com/install_aws_cli_ubuntu/
https://linuxhint.com/install_aws_cli_ubuntu/

[haooooooqi]

Hi,

Seems I am still seeing this issue when downloading 10M small files with multithread.

aws-cli/1.16.182 Python/3.8.10 Linux/5.4.0-81-generic botocore/1.12.172