Getting Connection closed with aiohttp client

[binrush]

Long story short

We have an asyncio-based application, which uses aiohttp to query REST api. Sometimes requests to this API fail with following error (pasted only aiohttp part of stack trace):

ClientConnectionError: Connection closed
(...)
  File "somemodule.py", line 101, in _request
    return await response.json(encoding='utf-8')
  File "aiohttp/client_reqrep.py", line 785, in json
    yield from self.read()
  File "aiohttp/client_reqrep.py", line 746, in read
    self._content = yield from self.content.read()
  File "aiohttp/streams.py", line 607, in read
    return (yield from super().read(n))
  File "aiohttp/streams.py", line 299, in read
    raise self._exception

This is easily reproduced when we generate a lot of api requests from our application, but it does not reproduced when we do the same requests with curl, so it does not seem to be server/network issue.

Expected behaviour

Client executes requests without errors (just like curl does)

Actual behaviour

client gets {{Connection closed}} error

Steps to reproduce

Do a lot of requests to specific server from specific application. Not sure how can it be reproduced in third-party environment.

Your environment

python 3.5.5
aiohttp 2.3.2 (also tested with 3.0.8 - same

additional information

I dumped network traffic using tcpdump and session with error looks totally the same as session without error (server ip changed to 10.10.0.1):

07:44:20.594548 IP 192.168.245.22.46280 > 69.175.3.9.https: Flags [S], seq 2283198217, win 29200, options [mss 1460,sackOK,TS val 266876113 ecr 0,nop,wscale 7], length 0
07:44:20.714665 IP 69.175.3.9.https > 192.168.245.22.46280: Flags [S.], seq 1753575808, ack 2283198218, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length
0
07:44:20.714825 IP 192.168.245.22.46280 > 10.10.0.1.https: Flags [.], ack 1, win 229, length 0
07:44:20.716552 IP 192.168.245.22.46280 > 10.10.0.1.https: Flags [P.], seq 1:518, ack 1, win 229, length 517
07:44:20.837448 IP 10.10.0.1.https > 192.168.245.22.46280: Flags [.], ack 518, win 123, length 0
07:44:20.839318 IP 10.10.0.1.https > 192.168.245.22.46280: Flags [P.], seq 1:2906, ack 518, win 123, length 2905
07:44:20.839375 IP 192.168.245.22.46280 > 10.10.0.1.https: Flags [.], ack 2906, win 274, length 0
07:44:20.842816 IP 192.168.245.22.46280 > 10.10.0.1.https: Flags [P.], seq 518:644, ack 2906, win 274, length 126
07:44:20.963454 IP 10.10.0.1.https > 192.168.245.22.46280: Flags [P.], seq 2906:3180, ack 644, win 123, length 274
07:44:20.965773 IP 192.168.245.22.46280 > 10.10.0.1.https: Flags [P.], seq 644:954, ack 3180, win 297, length 310
07:44:21.113806 IP 10.10.0.1.https > 192.168.245.22.46280: Flags [.], seq 3180:4640, ack 954, win 131, length 1460
07:44:21.117098 IP 10.10.0.1.https > 192.168.245.22.46280: Flags [P.], seq 4640:4643, ack 954, win 131, length 3
07:44:21.117161 IP 192.168.245.22.46280 > 10.10.0.1.https: Flags [.], ack 4643, win 320, length 0
07:44:21.117306 IP 10.10.0.1.https > 192.168.245.22.46280: Flags [P.], seq 4643:4677, ack 954, win 131, length 34
07:44:21.117841 IP 192.168.245.22.46280 > 10.10.0.1.https: Flags [P.], seq 954:985, ack 4677, win 320, length 31
07:44:21.238000 IP 10.10.0.1.https > 192.168.245.22.46280: Flags [F.], seq 4677, ack 985, win 131, length 0
07:44:21.239017 IP 192.168.245.22.46280 > 10.10.0.1.https: Flags [F.], seq 985, ack 4678, win 320, length 0
07:44:21.359176 IP 10.10.0.1.https > 192.168.245.22.46280: Flags [.], ack 986, win 131, length 0

[asvetlov]

Sorry, but if there is no way to reproduce the problem on depveloper's machine -- a chance to get is fixed is close to zero.

You wrote that aiohttp client generates closed by peer exceptions but curl never does. How did you make sure that curl processed all requests correctly and did it under the same load as aiohttp?

[binrush]

Thank you for your response. I understand that my description of problem is not enough for developers to understand the cause. However, probably you can suggest, how can we debug/investigate this issue to get more information on it?

[danielrobbins]

Please confirm that you are not running into the "abort HTTP requests that last longer than 5 minutes by default" issue: #2249

[binrush]

It's definitely not the same problem. Our requests are far less then 5 minutes, it's just 1K json + headers.

[binrush]

Hi. We found a problem in our own code: connection was closed before response was read (response was read outside of context manager). This issue can be closed

Sorry for taking your time, and thanks a lot for your work!

[asvetlov]

Cool!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a [new issue] for related bugs.
If you feel like there's important points made in this discussion, please include those exceprts into that [new issue].
[new issue]: https://github.com/aio-libs/aiohttp/issues/new