aiohttp>2.0.3 streaming api critical bug

[Rybak5611]

Long story short

Critical bug, affecting aiohttp>2.0.3

Expected behaviour

Reading page/file content line by line with

async for Line in Response.content

Actual behaviour

Loop hangs until interpreter raises concurrent.futures._base.TimeoutError error

Steps to reproduce

import aiohttp
import asyncio

async def Init():

	Response = await aiohttp.request( "GET", "http://www.google.com/" )

	async for Line in Response.content:
		print( Line )

if __name__ == "__main__":

	Loop = asyncio.get_event_loop()
	Loop.run_until_complete( Init() )

Your environment

Tested on Windows 10 build 15063, Debian 8
Python 3.6
aiohttp>2.0.3

Traceback ( aiohttp 2.1.0 )

PS C:\Users\root\Desktop> pip install -q aiohttp==2.1.0
PS C:\Users\root\Desktop> .\test.py
Traceback (most recent call last):
  File "C:\Users\root\Desktop\test.py", line 16, in <module>
    Loop.run_until_complete( Init() )
  File "C:\Py36\lib\asyncio\base_events.py", line 466, in run_until_complete
    return future.result()
  File "C:\Users\root\Desktop\test.py", line 10, in Init
    async for Line in Response.content:
  File "C:\Py36\lib\site-packages\aiohttp\streams.py", line 36, in __anext__
    rv = yield from self.read_func()
  File "C:\Py36\lib\site-packages\aiohttp\streams.py", line 570, in readline
    return (yield from super().readline())
  File "C:\Py36\lib\site-packages\aiohttp\streams.py", line 268, in readline
    yield from self._wait('readline')
  File "C:\Py36\lib\site-packages\aiohttp\streams.py", line 235, in _wait
    yield from waiter
  File "C:\Py36\lib\site-packages\aiohttp\helpers.py", line 706, in __exit__
    raise asyncio.TimeoutError from None
concurrent.futures._base.TimeoutError

[asvetlov]

Cannot reproduce.
Your example works as expected.

[hustshawn]

@asvetlov @Rybak5611
I tried to rewrite your example with below code, and use requests to get the page as an baseline. Unexpected result happen, and just stuck there after some while.

import aiohttp
import asyncio
import sys
import requests

async def Init():

    url = "http://www.dabeaz.com/"
    Response = await aiohttp.request( "GET", url )
    r = requests.get(url)
    print(r.text)
    print("###############")
    async for Line in Response.content:
        print( Line )

if __name__ == "__main__":

    Loop = asyncio.get_event_loop()
    Loop.run_until_complete( Init() )
    sys.exit(0)

The result as below: https://gist.github.com/asvetlov/0cc167d03fd1e77516d67e23fa8d1ebc

[Rybak5611]

As i wrote this affects only aiohttp>2.0.3 and only interation over content.
Versions 2.0.1, 2.0.2, 2.0.3 work fine and await ( await aiohttp.request( "GET" ) ).content() works on all versions

My guess is that it has smth to do with \r\n ending of each line in HTTP body.

Also this bug semi-reproducible, may also depend on network connection and OS.

[alexmnazarenko]

Hangs and timeouts for me too.

Environment: ubuntu 16.04, python 3.6.0, aiohttp 2.1.0

[asvetlov]

google.com works fine but http://www.dabeaz.com/ stucks on my laptop, so I've reproduced the issue.

My guess is that it has smth to do with \r\n ending of each line in HTTP body.
Also this bug semi-reproducible, may also depend on network connection and OS.

Maybe the bug is flaky and depends on many conditions (not including OS/network but also server's reverse proxy setup etc.)

@fafhrd91 I did not investigate it further. Would you take a look? The problem is really very serious and it is scaring me. I thought aiohttp client is very stable.

[fafhrd91]

Ok, will during weekend

[fafhrd91]

should be fixed in master, we have to make release

@asvetlov I added comment that explain the problem. but technically normal session usage can be affected with the same bug.

here is example:

    def get(url):
        return ClientSession().get(url)

proper fix for this bug would be to store reference to session in ClientResponse object. then we can delete ClientSession.detach method

[asvetlov]

Keeping in response a reference to session looks pretty fine to me.

[fafhrd91]

could you work on fix?

[asvetlov]

Sure

[marco-lavagnino]

I was having a similar issue (timeout errors) using this code:

with aiohttp.ClientSession(loop=asyncio.get_event_loop()) as session:
    response = await session.get("...")

chunk = await response.content.readany()

Using session without a with statement solved the issue:

session = aiohttp.ClientSession(loop=asyncio.get_event_loop())
response = await session.get("...")
chunk = await response.content.readany()

Maybe it's not the same problem, but this comment might help somebody.

[asvetlov]

with session closes the session at exit from statement block, next read from response fails obviously.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a [new issue] for related bugs.
If you feel like there's important points made in this discussion, please include those exceprts into that [new issue].
[new issue]: https://github.com/aio-libs/aiohttp/issues/new