Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Client support for s3gof3r #140

Closed
Licenser opened this issue Feb 19, 2014 · 8 comments
Closed

Client support for s3gof3r #140

Licenser opened this issue Feb 19, 2014 · 8 comments
Assignees
@yosukehara
Labels
Request S3-API
Milestone
1.0.1

Comments

@Licenser
Copy link
Contributor

s3gof3r is a very nice and fast s3 client in a quick test I got about 5-10x the speed against LoeFS compared to other clients since it takes advantage of parallel up and downloads.

So to get it work there were a few issues that I stumbled upon (probably not a complete list):

  • the hash check of eTAGs failed and had to be disabled
  • downloads get i/o timeouts on reads
  • downloading parts returns 200 (in LeoFS) but it seems a 206 is expected
  • no way to disable https on the Cleint (not a leo problem)

https://github.com/rlmcpherson/s3gof3r
@yosukehara
Copy link
Member

Thank you for your report. We'll try and check it.

@yosukehara
Copy link
Member

Quickly sharing: Fixed "downloading parts returns 200 (in LeoFS) but it seems a 206 is expected", which will be included in LeoFS v1.0.0-pre3.

@yosukehara yosukehara added this to the 1.0.0 milestone Feb 26, 2014
@yosukehara yosukehara self-assigned this Feb 26, 2014
@mocchira
Copy link
Member

mocchira commented Mar 5, 2014

multipart upload completion eTAG should be formatted as follows.

    ETag = MD5(Sum(p \in numberParts, MD5(PartBytes(p))) + "-" + numberParts

@mocchira
Copy link
Member

mocchira commented Mar 6, 2014

Hi @Licenser.
We just have fixed following issues you filed on dev branch.

  1. the hash check of eTAGs failed and had to be disabled
  2. downloading parts returns 200 (in LeoFS) but it seems a 206 is expected

But, we couldn't reproduce this issue.

  1. downloads get i/o timeouts on reads

At a glance,
since s3gof3r have used the fixed timeout value(https://github.com/rlmcpherson/s3gof3r/blob/master/s3gof3r.go#L48: 5sec)
under a very high network load or using narrowband, timeouts might happen.
So we'd think this issue is a s3gof3r problem as well as no way to disable https.
(Making timeout and url scheme configurable on s3gof3r will solve these two issues)

Anyway, thanks for sharing the great s3 client!

@Licenser
Copy link
Contributor Author

Licenser commented Mar 6, 2014

Hi @mocchira thank you so much :) this is awesome. I'll do some testing on the timeouts since I do tend to get them very frequently (after ~200MB) but it might be related to the setup bing 3 vm's on the same physical box. I'll have some extra systems end of the week and I can do some additional tests :)

@Licenser
Copy link
Contributor Author

A little update:

  • The 200 issue is solved! Thanks :)
  • Hashing for multipart seems still to be misbehaving
    MD5 hash of part hashes comparison failed. Hash from multipart complete header: 3fc6da276552fa204c079b51fdb71fd9". Calculated multipart hash: 3891444756702b8af28def4576d269de.

Timeouts happen like crazy:

Test parameters:

  • Network: 10GBit
  • GW: 16GB memory (both dev and 1.0.0pre3 have same results)
  • Storage: 64GB Memory, 60T disk (RAID 60 of spinning rust + SSD Cache)
  • Write Throughput: ~1GBit/s
  • Read Throughput (as per this test ~80 MBits/s) (I will open a separate issue for this since it seems quite bad)
bash$ gof3r get --endpoint=leofs -b bench -c 5 -s 5242880 -k 4g.2 -p down.2
2014/03/12 14:18:00 GET:  &{down.2 {4g.2 bench false 5 5242880 leofs false} }
2014/03/12 14:18:00 chunk total:  782
2014/03/12 14:18:00 content length :  4094722048
2014/03/12 14:18:00 concurrency:  5
2014/03/12 14:18:06 Error on attempt 0: retrying chunk: &{12 map[Range:[bytes=62914560-68157439] Date:[Wed, 12 Mar 2014 14:18:01 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:EyDM0WCaZLyNkfq9I1PKLfotSrE=]] 62914560 5242880 0xc2080e1f10 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:11 Error on attempt 1: retrying chunk: &{12 map[Range:[bytes=62914560-68157439] Date:[Wed, 12 Mar 2014 14:18:01 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:EyDM0WCaZLyNkfq9I1PKLfotSrE=]] 62914560 5242880 0xc2080e1f10 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:17 Error on attempt 0: retrying chunk: &{27 map[Range:[bytes=141557760-146800639] Date:[Wed, 12 Mar 2014 14:18:12 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:6Qr7fgfY3/S5opCHGqiV16TnPg8=]] 141557760 5242880 0xc2080e1ea0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:17 Error on attempt 0: retrying chunk: &{33 map[Authorization:[AWS ...:6Qr7fgfY3/S5opCHGqiV16TnPg8=] Range:[bytes=173015040-178257919] Date:[Wed, 12 Mar 2014 14:18:12 GMT] User-Agent:[S3Gof3r]] 173015040 5242880 0xc209fc6000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:17 Error on attempt 0: retrying chunk: &{37 map[Authorization:[AWS ...:6Qr7fgfY3/S5opCHGqiV16TnPg8=] Range:[bytes=193986560-199229439] Date:[Wed, 12 Mar 2014 14:18:12 GMT] User-Agent:[S3Gof3r]] 193986560 5242880 0xc2080e1f10 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:22 Error on attempt 0: retrying chunk: &{40 map[Date:[Wed, 12 Mar 2014 14:18:17 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:CGZDnpksdZr2jFI6oETVM6YijgY=] Range:[bytes=209715200-214958079]] 209715200 5242880 0xc209fc6000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:23 Error on attempt 0: retrying chunk: &{47 map[User-Agent:[S3Gof3r] Authorization:[AWS ...:ebhBDClB8IVzV4VmLu5yxrRviKE=] Range:[bytes=246415360-251658239] Date:[Wed, 12 Mar 2014 14:18:18 GMT]] 246415360 5242880 0xc209fc56c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:28 Error on attempt 0: retrying chunk: &{53 map[Range:[bytes=277872640-283115519] Date:[Wed, 12 Mar 2014 14:18:23 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:pzP7O2EekF5X9QYE1uvzCqsI/PY=]] 277872640 5242880 0xc209fc7c70 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:28 Error on attempt 0: retrying chunk: &{55 map[Range:[bytes=288358400-293601279] Date:[Wed, 12 Mar 2014 14:18:23 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:pzP7O2EekF5X9QYE1uvzCqsI/PY=]] 288358400 5242880 0xc2080e0000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:28 Error on attempt 0: retrying chunk: &{57 map[Range:[bytes=298844160-304087039] Date:[Wed, 12 Mar 2014 14:18:23 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:pzP7O2EekF5X9QYE1uvzCqsI/PY=]] 298844160 5242880 0xc209fc6000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:29 Error on attempt 0: retrying chunk: &{59 map[User-Agent:[S3Gof3r] Authorization:[AWS ...:tpvPHbQVcjHEDGuO4ZnRF44CLlw=] Range:[bytes=309329920-314572799] Date:[Wed, 12 Mar 2014 14:18:24 GMT]] 309329920 5242880 0xc2080e01c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:30 Error on attempt 0: retrying chunk: &{61 map[Range:[bytes=319815680-325058559] Date:[Wed, 12 Mar 2014 14:18:24 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:tpvPHbQVcjHEDGuO4ZnRF44CLlw=]] 319815680 5242880 0xc209fc4150 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:35 Error on attempt 0: retrying chunk: &{74 map[Range:[bytes=387973120-393215999] Date:[Wed, 12 Mar 2014 14:18:30 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:RhVcJ172Qnwh95pInDOu5NNgH8M=]] 387973120 5242880 0xc209fc55e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:36 Error on attempt 0: retrying chunk: &{76 map[Range:[bytes=398458880-403701759] Date:[Wed, 12 Mar 2014 14:18:30 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:RhVcJ172Qnwh95pInDOu5NNgH8M=]] 398458880 5242880 0xc2080e0000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:36 Error on attempt 0: retrying chunk: &{84 map[Date:[Wed, 12 Mar 2014 14:18:31 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:h1zZTI4xe1m8Egcw7WLJiTmiYRw=] Range:[bytes=440401920-445644799]] 440401920 5242880 0xc2080e1ea0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:41 Error on attempt 0: retrying chunk: &{87 map[User-Agent:[S3Gof3r] Authorization:[AWS ...:XJyTVEyx2LfqQ9mP09pPkL0uEHo=] Range:[bytes=456130560-461373439] Date:[Wed, 12 Mar 2014 14:18:36 GMT]] 456130560 5242880 0xc2080e00e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:47 Error on attempt 0: retrying chunk: &{103 map[User-Agent:[S3Gof3r] Authorization:[AWS ...:lbMd0MEChvpvwN8FGPpwGt+6NXM=] Range:[bytes=540016640-545259519] Date:[Wed, 12 Mar 2014 14:18:41 GMT]] 540016640 5242880 0xc2080e1f10 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:52 Error on attempt 0: retrying chunk: &{117 map[Range:[bytes=613416960-618659839] Date:[Wed, 12 Mar 2014 14:18:47 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:ujigLUsWP6BZUBhWTusN2L4ugvY=]] 613416960 5242880 0xc2080e01c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:52 Error on attempt 0: retrying chunk: &{119 map[Range:[bytes=623902720-629145599] Date:[Wed, 12 Mar 2014 14:18:47 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:ujigLUsWP6BZUBhWTusN2L4ugvY=]] 623902720 5242880 0xc209fc6000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:53 Error on attempt 0: retrying chunk: &{122 map[Range:[bytes=639631360-644874239] Date:[Wed, 12 Mar 2014 14:18:47 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:ujigLUsWP6BZUBhWTusN2L4ugvY=]] 639631360 5242880 0xc20a072000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:53 Error on attempt 0: retrying chunk: &{123 map[Date:[Wed, 12 Mar 2014 14:18:47 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:ujigLUsWP6BZUBhWTusN2L4ugvY=] Range:[bytes=644874240-650117119]] 644874240 5242880 0xc2080e02a0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:58 Error on attempt 0: retrying chunk: &{130 map[Range:[bytes=681574400-686817279] Date:[Wed, 12 Mar 2014 14:18:53 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:DZg6RnSsnoom1zLLdAcvkLweczw=]] 681574400 5242880 0xc2080e1f10 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:58 Error on attempt 0: retrying chunk: &{131 map[Authorization:[AWS ...:DZg6RnSsnoom1zLLdAcvkLweczw=] Range:[bytes=686817280-692060159] Date:[Wed, 12 Mar 2014 14:18:53 GMT] User-Agent:[S3Gof3r]] 686817280 5242880 0xc2080e00e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:58 Error on attempt 0: retrying chunk: &{132 map[Range:[bytes=692060160-697303039] Date:[Wed, 12 Mar 2014 14:18:53 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:DZg6RnSsnoom1zLLdAcvkLweczw=]] 692060160 5242880 0xc209fc56c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:58 Error on attempt 0: retrying chunk: &{133 map[User-Agent:[S3Gof3r] Authorization:[AWS ...:DZg6RnSsnoom1zLLdAcvkLweczw=] Range:[bytes=697303040-702545919] Date:[Wed, 12 Mar 2014 14:18:53 GMT]] 697303040 5242880 0xc2080e0000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:18:59 Error on attempt 0: retrying chunk: &{137 map[Range:[bytes=718274560-723517439] Date:[Wed, 12 Mar 2014 14:18:54 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:X//TP4lY8a3Cc9yjsZAJA0yN/FM=]] 718274560 5242880 0xc209fc7c70 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:04 Error on attempt 0: retrying chunk: &{144 map[Range:[bytes=754974720-760217599] Date:[Wed, 12 Mar 2014 14:18:58 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:wylUHoYmscZJs4UHojOHhEMFWjY=]] 754974720 5242880 0xc2080e02a0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:04 Error on attempt 0: retrying chunk: &{147 map[Date:[Wed, 12 Mar 2014 14:18:59 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:YrNlLSa+kLWqEuk+tg1m320MP1A=] Range:[bytes=770703360-775946239]] 770703360 5242880 0xc209fc55e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:09 Error on attempt 1: retrying chunk: &{144 map[Date:[Wed, 12 Mar 2014 14:18:58 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:wylUHoYmscZJs4UHojOHhEMFWjY=] Range:[bytes=754974720-760217599]] 754974720 5242880 0xc2080e02a0 0}, Error: read tcp leo:80: i/o timeout
        2014/03/12 14:19:14 Error on attempt 0: retrying chunk: &{156 map[Range:[bytes=817889280-823132159] Date:[Wed, 12 Mar 2014 14:19:09 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:OLRKxWBcz+gL7L5Nso00Lym6DVQ=]] 817889280 5242880 0xc2080e00e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:14 Error on attempt 0: retrying chunk: &{158 map[Range:[bytes=828375040-833617919] Date:[Wed, 12 Mar 2014 14:19:09 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:OLRKxWBcz+gL7L5Nso00Lym6DVQ=]] 828375040 5242880 0xc209fc56c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:14 Error on attempt 0: retrying chunk: &{162 map[Range:[bytes=849346560-854589439] Date:[Wed, 12 Mar 2014 14:19:09 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:OLRKxWBcz+gL7L5Nso00Lym6DVQ=]] 849346560 5242880 0xc2080e1ea0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:14 Error on attempt 0: retrying chunk: &{161 map[Range:[bytes=844103680-849346559] Date:[Wed, 12 Mar 2014 14:19:09 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:OLRKxWBcz+gL7L5Nso00Lym6DVQ=]] 844103680 5242880 0xc209fc7c70 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:20 Error on attempt 0: retrying chunk: &{167 map[Range:[bytes=875560960-880803839] Date:[Wed, 12 Mar 2014 14:19:15 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:Hy1EvPWt6ggpxE+n+AdYikXYAzQ=]] 875560960 5242880 0xc2080e1ea0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:20 Error on attempt 0: retrying chunk: &{171 map[Range:[bytes=896532480-901775359] Date:[Wed, 12 Mar 2014 14:19:15 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:Hy1EvPWt6ggpxE+n+AdYikXYAzQ=]] 896532480 5242880 0xc2080e1f10 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:25 Error on attempt 0: retrying chunk: &{179 map[Range:[bytes=938475520-943718399] Date:[Wed, 12 Mar 2014 14:19:20 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:Pg4Ep9uCPG1azvsREN5vqLMZ+l4=]] 938475520 5242880 0xc209fc7c70 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:25 Error on attempt 0: retrying chunk: &{184 map[Range:[bytes=964689920-969932799] Date:[Wed, 12 Mar 2014 14:19:20 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:Pg4Ep9uCPG1azvsREN5vqLMZ+l4=]] 964689920 5242880 0xc209fc4150 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:31 Error on attempt 0: retrying chunk: &{195 map[Range:[bytes=1022361600-1027604479] Date:[Wed, 12 Mar 2014 14:19:26 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:M4e0W5rr11AiDtHEk6uK7442mbI=]] 1022361600 5242880 0xc2080e1ea0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:31 Error on attempt 0: retrying chunk: &{196 map[User-Agent:[S3Gof3r] Authorization:[AWS ...:M4e0W5rr11AiDtHEk6uK7442mbI=] Range:[bytes=1027604480-1032847359] Date:[Wed, 12 Mar 2014 14:19:26 GMT]] 1027604480 5242880 0xc2080e0000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:31 Error on attempt 0: retrying chunk: &{198 map[Authorization:[AWS ...:M4e0W5rr11AiDtHEk6uK7442mbI=] Range:[bytes=1038090240-1043333119] Date:[Wed, 12 Mar 2014 14:19:26 GMT] User-Agent:[S3Gof3r]] 1038090240 5242880 0xc209fc6000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:37 Error on attempt 0: retrying chunk: &{213 map[Range:[bytes=1116733440-1121976319] Date:[Wed, 12 Mar 2014 14:19:32 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:E/r1xo08ZsqTtCGO+BdBSaOxHlw=]] 1116733440 5242880 0xc209fc7c70 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:37 Error on attempt 0: retrying chunk: &{212 map[Range:[bytes=1111490560-1116733439] Date:[Wed, 12 Mar 2014 14:19:32 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:E/r1xo08ZsqTtCGO+BdBSaOxHlw=]] 1111490560 5242880 0xc2080e00e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:37 Error on attempt 0: retrying chunk: &{217 map[Range:[bytes=1137704960-1142947839] Date:[Wed, 12 Mar 2014 14:19:32 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:E/r1xo08ZsqTtCGO+BdBSaOxHlw=]] 1137704960 5242880 0xc209fc56c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:42 Error on attempt 1: retrying chunk: &{213 map[Range:[bytes=1116733440-1121976319] Date:[Wed, 12 Mar 2014 14:19:32 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:E/r1xo08ZsqTtCGO+BdBSaOxHlw=]] 1116733440 5242880 0xc209fc7c70 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:43 Error on attempt 1: retrying chunk: &{217 map[Range:[bytes=1137704960-1142947839] Date:[Wed, 12 Mar 2014 14:19:32 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:E/r1xo08ZsqTtCGO+BdBSaOxHlw=]] 1137704960 5242880 0xc209fc56c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:48 Error on attempt 2: retrying chunk: &{213 map[Date:[Wed, 12 Mar 2014 14:19:32 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:E/r1xo08ZsqTtCGO+BdBSaOxHlw=] Range:[bytes=1116733440-1121976319]] 1116733440 5242880 0xc209fc7c70 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:48 Error on attempt 2: retrying chunk: &{217 map[User-Agent:[S3Gof3r] Authorization:[AWS ...:E/r1xo08ZsqTtCGO+BdBSaOxHlw=] Range:[bytes=1137704960-1142947839] Date:[Wed, 12 Mar 2014 14:19:32 GMT]] 1137704960 5242880 0xc209fc56c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:53 Error on attempt 3: retrying chunk: &{217 map[Range:[bytes=1137704960-1142947839] Date:[Wed, 12 Mar 2014 14:19:32 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:E/r1xo08ZsqTtCGO+BdBSaOxHlw=]] 1137704960 5242880 0xc209fc56c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:54 Error on attempt 0: retrying chunk: &{225 map[Authorization:[AWS ...:5qPaVcWCrtoEOburdDnJ275v1Ss=] Range:[bytes=1179648000-1184890879] Date:[Wed, 12 Mar 2014 14:19:49 GMT] User-Agent:[S3Gof3r]] 1179648000 5242880 0xc209fc55e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:54 Error on attempt 0: retrying chunk: &{226 map[Range:[bytes=1184890880-1190133759] Date:[Wed, 12 Mar 2014 14:19:49 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:5qPaVcWCrtoEOburdDnJ275v1Ss=]] 1184890880 5242880 0xc209fc7c70 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:54 Error on attempt 0: retrying chunk: &{224 map[Range:[bytes=1174405120-1179647999] Date:[Wed, 12 Mar 2014 14:19:49 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:5qPaVcWCrtoEOburdDnJ275v1Ss=]] 1174405120 5242880 0xc2080e01c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:19:59 Error on attempt 0: retrying chunk: &{233 map[Authorization:[AWS ...:81zINBwAlsXlmN70u1XFGJROCik=] Range:[bytes=1221591040-1226833919] Date:[Wed, 12 Mar 2014 14:19:54 GMT] User-Agent:[S3Gof3r]] 1221591040 5242880 0xc209fc4150 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:20:06 Error on attempt 0: retrying chunk: &{246 map[Authorization:[AWS ...:Tmn5Hgj4CfXWiNidRrmA4BayQAA=] Range:[bytes=1289748480-1294991359] Date:[Wed, 12 Mar 2014 14:20:00 GMT] User-Agent:[S3Gof3r]] 1289748480 5242880 0xc209fc7c70 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:20:11 Error on attempt 0: retrying chunk: &{260 map[Range:[bytes=1363148800-1368391679] Date:[Wed, 12 Mar 2014 14:20:06 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:IwgWriOAm2mnXYODnveD9/PBK2g=]] 1363148800 5242880 0xc20a072000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:20:16 Error on attempt 1: retrying chunk: &{260 map[Authorization:[AWS ...:IwgWriOAm2mnXYODnveD9/PBK2g=] Range:[bytes=1363148800-1368391679] Date:[Wed, 12 Mar 2014 14:20:06 GMT] User-Agent:[S3Gof3r]] 1363148800 5242880 0xc20a072000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:20:22 Error on attempt 0: retrying chunk: &{274 map[Range:[bytes=1436549120-1441791999] Date:[Wed, 12 Mar 2014 14:20:17 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:yMUM9d+prEnx6tWzDaJbayAIMgo=]] 1436549120 5242880 0xc2080e01c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:20:22 Error on attempt 0: retrying chunk: &{275 map[User-Agent:[S3Gof3r] Authorization:[AWS ...:yMUM9d+prEnx6tWzDaJbayAIMgo=] Range:[bytes=1441792000-1447034879] Date:[Wed, 12 Mar 2014 14:20:17 GMT]] 1441792000 5242880 0xc209fc6000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:20:28 Error on attempt 0: retrying chunk: &{287 map[Range:[bytes=1504706560-1509949439] Date:[Wed, 12 Mar 2014 14:20:23 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:UgiHB7sNzD8h5miPYXSsJiBksyU=]] 1504706560 5242880 0xc209fc55e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:20:34 Error on attempt 0: retrying chunk: &{299 map[Range:[bytes=1567621120-1572863999] Date:[Wed, 12 Mar 2014 14:20:29 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:XP1XLYQ2ME2GEiwxPGmXelR0NQ4=]] 1567621120 5242880 0xc2080e01c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:20:34 Error on attempt 0: retrying chunk: &{302 map[Authorization:[AWS ...:XP1XLYQ2ME2GEiwxPGmXelR0NQ4=] Range:[bytes=1583349760-1588592639] Date:[Wed, 12 Mar 2014 14:20:29 GMT] User-Agent:[S3Gof3r]] 1583349760 5242880 0xc2080e00e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:20:35 Error on attempt 0: retrying chunk: &{307 map[Range:[bytes=1609564160-1614807039] Date:[Wed, 12 Mar 2014 14:20:29 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:XP1XLYQ2ME2GEiwxPGmXelR0NQ4=]] 1609564160 5242880 0xc209fc56c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:20:41 Error on attempt 0: retrying chunk: &{319 map[User-Agent:[S3Gof3r] Authorization:[AWS ...:hwhMOY3b0VbZXEsIJf/Ut1q9ays=] Range:[bytes=1672478720-1677721599] Date:[Wed, 12 Mar 2014 14:20:35 GMT]] 1672478720 5242880 0xc2080e1f10 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:20:41 Error on attempt 0: retrying chunk: &{320 map[Range:[bytes=1677721600-1682964479] Date:[Wed, 12 Mar 2014 14:20:36 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:3KqPY2NXxQkuW4IzEhdoPKIR6eM=]] 1677721600 5242880 0xc2080e00e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:20:41 Error on attempt 0: retrying chunk: &{322 map[Range:[bytes=1688207360-1693450239] Date:[Wed, 12 Mar 2014 14:20:36 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:3KqPY2NXxQkuW4IzEhdoPKIR6eM=]] 1688207360 5242880 0xc209fc4150 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:20:41 Error on attempt 0: retrying chunk: &{323 map[Range:[bytes=1693450240-1698693119] Date:[Wed, 12 Mar 2014 14:20:36 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:3KqPY2NXxQkuW4IzEhdoPKIR6eM=]] 1693450240 5242880 0xc2080e01c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:20:42 Error on attempt 0: retrying chunk: &{325 map[Date:[Wed, 12 Mar 2014 14:20:36 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:3KqPY2NXxQkuW4IzEhdoPKIR6eM=] Range:[bytes=1703936000-1709178879]] 1703936000 5242880 0xc2080e0000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:20:46 Error on attempt 1: retrying chunk: &{319 map[Range:[bytes=1672478720-1677721599] Date:[Wed, 12 Mar 2014 14:20:35 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:hwhMOY3b0VbZXEsIJf/Ut1q9ays=]] 1672478720 5242880 0xc2080e1f10 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:20:46 Error on attempt 1: retrying chunk: &{320 map[Range:[bytes=1677721600-1682964479] Date:[Wed, 12 Mar 2014 14:20:36 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:3KqPY2NXxQkuW4IzEhdoPKIR6eM=]] 1677721600 5242880 0xc2080e00e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:20:47 Error on attempt 0: retrying chunk: &{326 map[Date:[Wed, 12 Mar 2014 14:20:42 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:C4G/8VliqRTugZ4WkD18kIg9H9s=] Range:[bytes=1709178880-1714421759]] 1709178880 5242880 0xc2080e02a0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:20:53 Error on attempt 0: retrying chunk: &{331 map[User-Agent:[S3Gof3r] Authorization:[AWS ...:tnElcSMTcBXMFKVwdab1qRefEys=] Range:[bytes=1735393280-1740636159] Date:[Wed, 12 Mar 2014 14:20:47 GMT]] 1735393280 5242880 0xc209fc55e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:20:53 Error on attempt 0: retrying chunk: &{336 map[Date:[Wed, 12 Mar 2014 14:20:48 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:ke2wNAm0b9JlgdMq+NE0prWdww4=] Range:[bytes=1761607680-1766850559]] 1761607680 5242880 0xc209fc7c70 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:01 Error on attempt 0: retrying chunk: &{354 map[Range:[bytes=1855979520-1861222399] Date:[Wed, 12 Mar 2014 14:20:56 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:DUWFn/zf4MHkPWJgLrSJBnfbj+E=]] 1855979520 5242880 0xc2080e1f10 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:07 Error on attempt 0: retrying chunk: &{368 map[Range:[bytes=1929379840-1934622719] Date:[Wed, 12 Mar 2014 14:21:02 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:0kLeP/9Nwv+li+BYsoOEY5T+0gY=]] 1929379840 5242880 0xc209fc6000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:08 Error on attempt 0: retrying chunk: &{370 map[Authorization:[AWS ...:0kLeP/9Nwv+li+BYsoOEY5T+0gY=] Range:[bytes=1939865600-1945108479] Date:[Wed, 12 Mar 2014 14:21:02 GMT] User-Agent:[S3Gof3r]] 1939865600 5242880 0xc20a072000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:13 Error on attempt 1: retrying chunk: &{368 map[Date:[Wed, 12 Mar 2014 14:21:02 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:0kLeP/9Nwv+li+BYsoOEY5T+0gY=] Range:[bytes=1929379840-1934622719]] 1929379840 5242880 0xc209fc6000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:18 Error on attempt 0: retrying chunk: &{381 map[Date:[Wed, 12 Mar 2014 14:21:13 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:GqQVK0RPCvmSJZq4l4V1POUSxaU=] Range:[bytes=1997537280-2002780159]] 1997537280 5242880 0xc2080e1ea0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:24 Error on attempt 0: retrying chunk: &{392 map[Range:[bytes=2055208960-2060451839] Date:[Wed, 12 Mar 2014 14:21:19 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:eltvFZlNd1zmc8BAbg23ccnAEGk=]] 2055208960 5242880 0xc209fc4150 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:24 Error on attempt 0: retrying chunk: &{396 map[Range:[bytes=2076180480-2081423359] Date:[Wed, 12 Mar 2014 14:21:19 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:eltvFZlNd1zmc8BAbg23ccnAEGk=]] 2076180480 5242880 0xc209fc56c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:24 Error on attempt 0: retrying chunk: &{397 map[Date:[Wed, 12 Mar 2014 14:21:19 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:eltvFZlNd1zmc8BAbg23ccnAEGk=] Range:[bytes=2081423360-2086666239]] 2081423360 5242880 0xc209fc6000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:29 Error on attempt 1: retrying chunk: &{392 map[User-Agent:[S3Gof3r] Authorization:[AWS ...:eltvFZlNd1zmc8BAbg23ccnAEGk=] Range:[bytes=2055208960-2060451839] Date:[Wed, 12 Mar 2014 14:21:19 GMT]] 2055208960 5242880 0xc209fc4150 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:36 Error on attempt 0: retrying chunk: &{408 map[Range:[bytes=2139095040-2144337919] Date:[Wed, 12 Mar 2014 14:21:31 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:EZ0mCFyxOVRKUyHxJpGTfm2LSx4=]] 2139095040 5242880 0xc2080e01c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:37 Error on attempt 0: retrying chunk: &{415 map[Authorization:[AWS ...:ElhP1au+XcwwJfx42maDBN7n8gA=] Range:[bytes=2175795200-2181038079] Date:[Wed, 12 Mar 2014 14:21:32 GMT] User-Agent:[S3Gof3r]] 2175795200 5242880 0xc2080e00e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:43 Error on attempt 0: retrying chunk: &{420 map[Range:[bytes=2202009600-2207252479] Date:[Wed, 12 Mar 2014 14:21:37 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:BcmpB5U9IB64wQ9eLHEFl54y7VM=]] 2202009600 5242880 0xc2080e1ea0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:43 Error on attempt 0: retrying chunk: &{422 map[Range:[bytes=2212495360-2217738239] Date:[Wed, 12 Mar 2014 14:21:37 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:BcmpB5U9IB64wQ9eLHEFl54y7VM=]] 2212495360 5242880 0xc209fc6000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:43 Error on attempt 0: retrying chunk: &{423 map[Range:[bytes=2217738240-2222981119] Date:[Wed, 12 Mar 2014 14:21:37 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:BcmpB5U9IB64wQ9eLHEFl54y7VM=]] 2217738240 5242880 0xc209fc56c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:49 Error on attempt 0: retrying chunk: &{431 map[Range:[bytes=2259681280-2264924159] Date:[Wed, 12 Mar 2014 14:21:43 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:9APxLcEvUODWMKH4m0Qh5BydYrg=]] 2259681280 5242880 0xc20a072000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:49 Error on attempt 0: retrying chunk: &{432 map[Authorization:[AWS ...:9APxLcEvUODWMKH4m0Qh5BydYrg=] Range:[bytes=2264924160-2270167039] Date:[Wed, 12 Mar 2014 14:21:43 GMT] User-Agent:[S3Gof3r]] 2264924160 5242880 0xc2080e0000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:49 Error on attempt 0: retrying chunk: &{436 map[User-Agent:[S3Gof3r] Authorization:[AWS ...:uCd3fa3bxwGTW2nnghfWR5iODe4=] Range:[bytes=2285895680-2291138559] Date:[Wed, 12 Mar 2014 14:21:44 GMT]] 2285895680 5242880 0xc209fc55e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:50 Error on attempt 0: retrying chunk: &{438 map[Authorization:[AWS ...:uCd3fa3bxwGTW2nnghfWR5iODe4=] Range:[bytes=2296381440-2301624319] Date:[Wed, 12 Mar 2014 14:21:44 GMT] User-Agent:[S3Gof3r]] 2296381440 5242880 0xc209fc56c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:50 Error on attempt 0: retrying chunk: &{441 map[Date:[Wed, 12 Mar 2014 14:21:45 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:GMYQZTeElQ/hNAGfOyFoElnxRKE=] Range:[bytes=2312110080-2317352959]] 2312110080 5242880 0xc2080e1ea0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:21:56 Error on attempt 0: retrying chunk: &{448 map[Authorization:[AWS ...:rjoqyGinklrDe0HAXAoKdxOiixA=] Range:[bytes=2348810240-2354053119] Date:[Wed, 12 Mar 2014 14:21:50 GMT] User-Agent:[S3Gof3r]] 2348810240 5242880 0xc2080e1f10 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:02 Error on attempt 0: retrying chunk: &{461 map[Range:[bytes=2416967680-2422210559] Date:[Wed, 12 Mar 2014 14:21:57 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:dwN/bDiLVQY88NGv0n3GMiONlhA=]] 2416967680 5242880 0xc2080e00e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:02 Error on attempt 0: retrying chunk: &{459 map[Range:[bytes=2406481920-2411724799] Date:[Wed, 12 Mar 2014 14:21:57 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:dwN/bDiLVQY88NGv0n3GMiONlhA=]] 2406481920 5242880 0xc2080e1ea0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:08 Error on attempt 1: retrying chunk: &{461 map[Range:[bytes=2416967680-2422210559] Date:[Wed, 12 Mar 2014 14:21:57 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:dwN/bDiLVQY88NGv0n3GMiONlhA=]] 2416967680 5242880 0xc2080e00e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:13 Error on attempt 2: retrying chunk: &{461 map[User-Agent:[S3Gof3r] Authorization:[AWS ...:dwN/bDiLVQY88NGv0n3GMiONlhA=] Range:[bytes=2416967680-2422210559] Date:[Wed, 12 Mar 2014 14:21:57 GMT]] 2416967680 5242880 0xc2080e00e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:20 Error on attempt 0: retrying chunk: &{476 map[Range:[bytes=2495610880-2500853759] Date:[Wed, 12 Mar 2014 14:22:14 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:hG9iLE8s3aalh5Ls1PSAFFzXEqo=]] 2495610880 5242880 0xc209fc6000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:26 Error on attempt 0: retrying chunk: &{492 map[Range:[bytes=2579496960-2584739839] Date:[Wed, 12 Mar 2014 14:22:21 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:/Ct+T5PSpf9sJmGYMv+lDqw78pY=]] 2579496960 5242880 0xc2080e1f10 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:26 Error on attempt 0: retrying chunk: &{489 map[Range:[bytes=2563768320-2569011199] Date:[Wed, 12 Mar 2014 14:22:21 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:/Ct+T5PSpf9sJmGYMv+lDqw78pY=]] 2563768320 5242880 0xc209fc4150 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:35 Error on attempt 0: retrying chunk: &{532 map[Range:[bytes=2789212160-2794455039] Date:[Wed, 12 Mar 2014 14:22:30 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:Yss6ixzf7JRNW6QuCqmlIaq9d8k=]] 2789212160 5242880 0xc2080e0000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:36 Error on attempt 0: retrying chunk: &{539 map[Range:[bytes=2825912320-2831155199] Date:[Wed, 12 Mar 2014 14:22:30 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:Yss6ixzf7JRNW6QuCqmlIaq9d8k=]] 2825912320 5242880 0xc2080e02a0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:41 Error on attempt 0: retrying chunk: &{544 map[Range:[bytes=2852126720-2857369599] Date:[Wed, 12 Mar 2014 14:22:36 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:wHEswPNkiyzuYz4HLCwLLb+CwHM=]] 2852126720 5242880 0xc209fc56c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:42 Error on attempt 0: retrying chunk: &{552 map[Range:[bytes=2894069760-2899312639] Date:[Wed, 12 Mar 2014 14:22:37 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:Cav1sc9lXem0UwISY4lVbPj8WAU=]] 2894069760 5242880 0xc2080e02a0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:47 Error on attempt 0: retrying chunk: &{557 map[Range:[bytes=2920284160-2925527039] Date:[Wed, 12 Mar 2014 14:22:42 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:qtJ6G5vPzfPS2YumFCNK+SIaXdU=]] 2920284160 5242880 0xc2080e01c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:47 Error on attempt 0: retrying chunk: &{559 map[Range:[bytes=2930769920-2936012799] Date:[Wed, 12 Mar 2014 14:22:42 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:qtJ6G5vPzfPS2YumFCNK+SIaXdU=]] 2930769920 5242880 0xc209fc7c70 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:48 Error on attempt 0: retrying chunk: &{560 map[User-Agent:[S3Gof3r] Authorization:[AWS ...:qtJ6G5vPzfPS2YumFCNK+SIaXdU=] Range:[bytes=2936012800-2941255679] Date:[Wed, 12 Mar 2014 14:22:42 GMT]] 2936012800 5242880 0xc2080e00e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:48 Error on attempt 0: retrying chunk: &{561 map[Date:[Wed, 12 Mar 2014 14:22:42 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:qtJ6G5vPzfPS2YumFCNK+SIaXdU=] Range:[bytes=2941255680-2946498559]] 2941255680 5242880 0xc2080e1f10 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:48 Error on attempt 0: retrying chunk: &{562 map[Range:[bytes=2946498560-2951741439] Date:[Wed, 12 Mar 2014 14:22:42 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:qtJ6G5vPzfPS2YumFCNK+SIaXdU=]] 2946498560 5242880 0xc20a072000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:53 Error on attempt 1: retrying chunk: &{559 map[Range:[bytes=2930769920-2936012799] Date:[Wed, 12 Mar 2014 14:22:42 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:qtJ6G5vPzfPS2YumFCNK+SIaXdU=]] 2930769920 5242880 0xc209fc7c70 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:59 Error on attempt 0: retrying chunk: &{571 map[Range:[bytes=2993684480-2998927359] Date:[Wed, 12 Mar 2014 14:22:53 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:cw9qxfjJxuB5r4k5yIPTcFzkZyM=]] 2993684480 5242880 0xc209fc55e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:59 Error on attempt 0: retrying chunk: &{570 map[Range:[bytes=2988441600-2993684479] Date:[Wed, 12 Mar 2014 14:22:53 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:cw9qxfjJxuB5r4k5yIPTcFzkZyM=]] 2988441600 5242880 0xc2080e02a0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:59 Error on attempt 0: retrying chunk: &{572 map[User-Agent:[S3Gof3r] Authorization:[AWS ...:cw9qxfjJxuB5r4k5yIPTcFzkZyM=] Range:[bytes=2998927360-3004170239] Date:[Wed, 12 Mar 2014 14:22:53 GMT]] 2998927360 5242880 0xc2080e0000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:22:59 Error on attempt 0: retrying chunk: &{578 map[Range:[bytes=3030384640-3035627519] Date:[Wed, 12 Mar 2014 14:22:54 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:GxH4FSh5YhXHDBZ9+zA8qn1obiw=]] 3030384640 5242880 0xc2080e1f10 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:23:04 Error on attempt 1: retrying chunk: &{571 map[Range:[bytes=2993684480-2998927359] Date:[Wed, 12 Mar 2014 14:22:53 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:cw9qxfjJxuB5r4k5yIPTcFzkZyM=]] 2993684480 5242880 0xc209fc55e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:23:10 Error on attempt 0: retrying chunk: &{585 map[Range:[bytes=3067084800-3072327679] Date:[Wed, 12 Mar 2014 14:23:05 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:mkeH9RJ0jEOX+9mhk59LZVjjeVg=]] 3067084800 5242880 0xc20a072000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:23:11 Error on attempt 0: retrying chunk: &{594 map[Authorization:[AWS ...:qNYZVF7Vu/0eq/pXnNtAcl/KKU8=] Range:[bytes=3114270720-3119513599] Date:[Wed, 12 Mar 2014 14:23:06 GMT] User-Agent:[S3Gof3r]] 3114270720 5242880 0xc209fc55e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:23:17 Error on attempt 0: retrying chunk: &{599 map[Range:[bytes=3140485120-3145727999] Date:[Wed, 12 Mar 2014 14:23:11 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:1NOX6sy5QipBkLlD3Wk8wiOuUTw=]] 3140485120 5242880 0xc2080e00e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:23:18 Error on attempt 0: retrying chunk: &{608 map[Range:[bytes=3187671040-3192913919] Date:[Wed, 12 Mar 2014 14:23:13 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:n5452CET2m/BScZFCiL3NeMXa4U=]] 3187671040 5242880 0xc2080e02a0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:23:19 Error on attempt 0: retrying chunk: &{605 map[Range:[bytes=3171942400-3177185279] Date:[Wed, 12 Mar 2014 14:23:13 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:n5452CET2m/BScZFCiL3NeMXa4U=]] 3171942400 5242880 0xc209fc6000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:23:26 Error on attempt 0: retrying chunk: &{618 map[Authorization:[AWS ...:1VgqcY1TJuLoQ52IYpW8x8rkQLE=] Range:[bytes=3240099840-3245342719] Date:[Wed, 12 Mar 2014 14:23:20 GMT] User-Agent:[S3Gof3r]] 3240099840 5242880 0xc2080e01c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:23:26 Error on attempt 0: retrying chunk: &{620 map[Date:[Wed, 12 Mar 2014 14:23:20 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:1VgqcY1TJuLoQ52IYpW8x8rkQLE=] Range:[bytes=3250585600-3255828479]] 3250585600 5242880 0xc2080e00e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:23:27 Error on attempt 0: retrying chunk: &{629 map[Range:[bytes=3297771520-3303014399] Date:[Wed, 12 Mar 2014 14:23:22 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:e/8C+AlGX3u0QSo9rxoXa6CJHnI=]] 3297771520 5242880 0xc209fc6000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:23:32 Error on attempt 0: retrying chunk: &{631 map[Range:[bytes=3308257280-3313500159] Date:[Wed, 12 Mar 2014 14:23:27 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:8TenSuOvOCkkoWH/34MfRIk2hEw=]] 3308257280 5242880 0xc2080e1f10 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:23:33 Error on attempt 0: retrying chunk: &{635 map[User-Agent:[S3Gof3r] Authorization:[AWS ...:8TenSuOvOCkkoWH/34MfRIk2hEw=] Range:[bytes=3329228800-3334471679] Date:[Wed, 12 Mar 2014 14:23:27 GMT]] 3329228800 5242880 0xc209fc7c70 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:23:40 Error on attempt 0: retrying chunk: &{648 map[Range:[bytes=3397386240-3402629119] Date:[Wed, 12 Mar 2014 14:23:34 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:OpblR7ON+57qm34Hdm5i7M5HoRE=]] 3397386240 5242880 0xc209fc7c70 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:23:40 Error on attempt 0: retrying chunk: &{654 map[Range:[bytes=3428843520-3434086399] Date:[Wed, 12 Mar 2014 14:23:35 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:ffSrJXEWvnDh2w+YpgWAfJXxwSM=]] 3428843520 5242880 0xc2080e02a0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:23:46 Error on attempt 0: retrying chunk: &{662 map[Range:[bytes=3470786560-3476029439] Date:[Wed, 12 Mar 2014 14:23:41 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:PDBqXJT7GINdrxy0jIs/i6n9XQs=]] 3470786560 5242880 0xc2080e1ea0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:23:46 Error on attempt 0: retrying chunk: &{661 map[User-Agent:[S3Gof3r] Authorization:[AWS ...:PDBqXJT7GINdrxy0jIs/i6n9XQs=] Range:[bytes=3465543680-3470786559] Date:[Wed, 12 Mar 2014 14:23:41 GMT]] 3465543680 5242880 0xc209fc4150 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:23:53 Error on attempt 0: retrying chunk: &{672 map[Range:[bytes=3523215360-3528458239] Date:[Wed, 12 Mar 2014 14:23:47 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:rbSyo/uDlCAeANje184qdwKtbsg=]] 3523215360 5242880 0xc209fc6000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:23:53 Error on attempt 0: retrying chunk: &{673 map[Range:[bytes=3528458240-3533701119] Date:[Wed, 12 Mar 2014 14:23:47 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:rbSyo/uDlCAeANje184qdwKtbsg=]] 3528458240 5242880 0xc2080e02a0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:23:53 Error on attempt 0: retrying chunk: &{675 map[Authorization:[AWS ...:rbSyo/uDlCAeANje184qdwKtbsg=] Range:[bytes=3538944000-3544186879] Date:[Wed, 12 Mar 2014 14:23:47 GMT] User-Agent:[S3Gof3r]] 3538944000 5242880 0xc2080e0000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:23:58 Error on attempt 1: retrying chunk: &{675 map[Authorization:[AWS ...:rbSyo/uDlCAeANje184qdwKtbsg=] Range:[bytes=3538944000-3544186879] Date:[Wed, 12 Mar 2014 14:23:47 GMT] User-Agent:[S3Gof3r]] 3538944000 5242880 0xc2080e0000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:24:05 Error on attempt 0: retrying chunk: &{683 map[User-Agent:[S3Gof3r] Authorization:[AWS ...:mlhoqbN2lkquMGqIigX5O8PcVBI=] Range:[bytes=3580887040-3586129919] Date:[Wed, 12 Mar 2014 14:24:00 GMT]] 3580887040 5242880 0xc20a072000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:24:10 Error on attempt 1: retrying chunk: &{683 map[Range:[bytes=3580887040-3586129919] Date:[Wed, 12 Mar 2014 14:24:00 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:mlhoqbN2lkquMGqIigX5O8PcVBI=]] 3580887040 5242880 0xc20a072000 0}, Error: read tcp leo:80: i/o timeout
`2014/03/12 14:24:17 Error on attempt 0: retrying chunk: &{702 map[Range:[bytes=3680501760-3685744639] Date:[Wed, 12 Mar 2014 14:24:12 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:ujn50Akj5uDZCSvUFOopgNzJsbM=]] 3680501760 5242880 0xc209fc6000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:24:25 Error on attempt 0: retrying chunk: &{729 map[Range:[bytes=3822059520-3827302399] Date:[Wed, 12 Mar 2014 14:24:20 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:8x8mRDJd42w9dqAMlYBOTimd8Ao=]] 3822059520 5242880 0xc209fc55e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:24:26 Error on attempt 0: retrying chunk: &{732 map[Range:[bytes=3837788160-3843031039] Date:[Wed, 12 Mar 2014 14:24:20 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:8x8mRDJd42w9dqAMlYBOTimd8Ao=]] 3837788160 5242880 0xc2080e01c0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:24:27 Error on attempt 0: retrying chunk: &{738 map[Range:[bytes=3869245440-3874488319] Date:[Wed, 12 Mar 2014 14:24:22 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:sExkpjISsdC4WLVjNi3hbL2C4vg=]] 3869245440 5242880 0xc2080e1ea0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:24:27 Error on attempt 0: retrying chunk: &{737 map[Range:[bytes=3864002560-3869245439] Date:[Wed, 12 Mar 2014 14:24:22 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:sExkpjISsdC4WLVjNi3hbL2C4vg=]] 3864002560 5242880 0xc20a072000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:24:33 Error on attempt 1: retrying chunk: &{738 map[Range:[bytes=3869245440-3874488319] Date:[Wed, 12 Mar 2014 14:24:22 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:sExkpjISsdC4WLVjNi3hbL2C4vg=]] 3869245440 5242880 0xc2080e1ea0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:24:33 Error on attempt 1: retrying chunk: &{737 map[Range:[bytes=3864002560-3869245439] Date:[Wed, 12 Mar 2014 14:24:22 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:sExkpjISsdC4WLVjNi3hbL2C4vg=]] 3864002560 5242880 0xc20a072000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:24:33 Error on attempt 0: retrying chunk: &{741 map[Range:[bytes=3884974080-3890216959] Date:[Wed, 12 Mar 2014 14:24:27 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:sILdJWybmtv5zv9VXYG5hyWhuws=]] 3884974080 5242880 0xc2080e0000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:24:40 Error on attempt 0: retrying chunk: &{754 map[Range:[bytes=3953131520-3958374399] Date:[Wed, 12 Mar 2014 14:24:35 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:pxsX7G7YjGDO59bf5VvP3kyjNcI=]] 3953131520 5242880 0xc2080e0000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:24:40 Error on attempt 0: retrying chunk: &{760 map[User-Agent:[S3Gof3r] Authorization:[AWS ...:pxsX7G7YjGDO59bf5VvP3kyjNcI=] Range:[bytes=3984588800-3989831679] Date:[Wed, 12 Mar 2014 14:24:35 GMT]] 3984588800 5242880 0xc2080e00e0 0}, Error: Get http://bench.ny4.storage.lucerahq.com/4g.2: net/http: timeout awaiting response headers
2014/03/12 14:24:41 Error on attempt 0: retrying chunk: &{758 map[Date:[Wed, 12 Mar 2014 14:24:35 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:pxsX7G7YjGDO59bf5VvP3kyjNcI=] Range:[bytes=3974103040-3979345919]] 3974103040 5242880 0xc2080e02a0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:24:41 Error on attempt 0: retrying chunk: &{761 map[Range:[bytes=3989831680-3995074559] Date:[Wed, 12 Mar 2014 14:24:35 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:pxsX7G7YjGDO59bf5VvP3kyjNcI=]] 3989831680 5242880 0xc2080e1f10 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:24:46 Error on attempt 1: retrying chunk: &{760 map[Range:[bytes=3984588800-3989831679] Date:[Wed, 12 Mar 2014 14:24:35 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:pxsX7G7YjGDO59bf5VvP3kyjNcI=]] 3984588800 5242880 0xc2080e00e0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:24:47 Error on attempt 0: retrying chunk: &{765 map[Range:[bytes=4010803200-4016046079] Date:[Wed, 12 Mar 2014 14:24:42 GMT] User-Agent:[S3Gof3r] Authorization:[AWS ...:puVxPnVNFvhtzIVOv62sprHuq2U=]] 4010803200 5242880 0xc2080e02a0 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:24:53 Error on attempt 0: retrying chunk: &{771 map[Authorization:[AWS ...:ibKn+DG92eytVQt5asqLKQ7BnT0=] Range:[bytes=4042260480-4047503359] Date:[Wed, 12 Mar 2014 14:24:47 GMT] User-Agent:[S3Gof3r]] 4042260480 5242880 0xc20a072000 0}, Error: read tcp leo:80: i/o timeout
2014/03/12 14:24:53 makes: 13
2014/03/12 14:24:53 md5:  d535ebcf56e726abffd6d0fb28ef4473
2014/03/12 14:24:53 md5Path:  .md5/4g.2.md5
2014/03/12 14:24:53 Headers:  map[Content-Type:[application/octet-stream] Etag:["d41d8cd98f00b204e9800998ecf8427e"] Content-Length:[4094722048] Last-Modified:[Wed, 12 Mar 2014 14:17:50 GMT] Date:[Wed, 12 Mar 2014 14:18:00 GMT] Server:[LeoFS]]
2014/03/12 14:24:53 Duration: 6m52.895093699s

@mocchira
Copy link
Member

  • Hashing for multipart seems still to be misbehaving
    MD5 hash of part hashes comparison failed. Hash from multipart complete header: >3fc6da276552fa204c079b51fdb71fd9". Calculated multipart hash: >3891444756702b8af28def4576d269de.

We tested some test files like files generated by dd with /dev/urandom and some OS images published on the web but couldn't reproduce this issue.
So if possible could you share files you have used with us?

Timeouts happen like crazy:

Thanks to your contributions we finally got it.
Since the root cause is same with #154, details will be described on #154.

@yosukehara yosukehara modified the milestones: 1.0.1, 1.0.0 Apr 15, 2014
@yosukehara
Copy link
Member

So closed #154, I've closed this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yosukehara yosukehara

Labels
Request S3-API
Projects
None yet
Milestone
1.0.1
Development

No branches or pull requests
3 participants
@Licenser @yosukehara @mocchira