RuntimeError with aiobotocore + Tornado

[andymo]

Hopefully this is the right repo to post this issue to, but I suspect the issue may be from aiobotocore's calling of aiohttp.

The example code:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from asyncio import get_event_loop
from datetime import timedelta

import aiobotocore
from tornado.gen import with_timeout, convert_yielded
from tornado.httpserver import HTTPServer
from tornado.platform.asyncio import AsyncIOMainLoop
from tornado.web import Application
from tornado.web import RequestHandler


class Handler(RequestHandler):

    async def get(self):
        session = aiobotocore.get_session()
        client = session.create_client('s3')

        response = await client.list_buckets()

        self.write(response)


if __name__ == '__main__':
    AsyncIOMainLoop().install()

    app = Application([
        (r'/', Handler),
    ])
    server = HTTPServer(app)
    server.listen(9999)

    get_event_loop().run_forever()

The full exception:

Uncaught exception GET / (127.0.0.1)
HTTPServerRequest(protocol='http', host='localhost:9999', method='GET', uri='/', version='HTTP/1.1', remote_ip='127.0.0.1', headers={'Accept': '*/*', 'User-Agent': 'curl/7.43.0', 'Host': 'localhost:9999'})
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/web.py", line 1469, in _execute
    result = yield result
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/gen.py", line 285, in wrapper
    yielded = next(result)
  File "<string>", line 6, in _wrap_awaitable
  File "test_aiobotocore.py", line 20, in get
    response = await client.list_buckets()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiobotocore/client.py", line 78, in _make_api_call
    operation_model, request_dict)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiobotocore/endpoint.py", line 211, in _send_request
    exception)):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiobotocore/endpoint.py", line 243, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/botocore/retryhandler.py", line 269, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiobotocore/endpoint.py", line 269, in _get_response
    request.method, request.url, request.headers, request.body)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiobotocore/endpoint.py", line 200, in _request
    allow_redirects=False)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/client.py", line 577, in __iter__
    resp = yield from self._coro
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/client.py", line 200, in _request
    with timer:
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/aiohttp/helpers.py", line 750, in __enter__
    raise RuntimeError('Timeout context manager should be used '
RuntimeError: Timeout context manager should be used inside a task
500 GET / (127.0.0.1) 68.54ms

I have read both aio-libs/aiohttp#1180 and aio-libs/aiohttp#877 which are most probably related to this issue.

This example code is a modification of the working example (which I tested works on my system before implementing aiobotocore) provided in issue 1180.

I also tried adding the with_timeout and convert_yielded wrappers as suggested in 877 but I receive the same error.

I know that 1180 ended up in allowing timeout=None to be passed to a client request, perhaps something similar can be implemented here that eventually gets passed to the client session request?

System information:

Python 3.5.2
aiobotocore (0.2.1)
aiohttp (1.3.3)
botocore (1.5.0)
tornado (4.4.2)

[thehesiod]

from what I see this is an issue with your usecase/tornado. aiobotcore and its dependencies require the use of an Event loop that's compatible with BaseEventLoop. The BaseEventLoop requires support of Tasks and a task requires a current_task and it doesn't seem like tornado sets/provides this. btw even passing timeout=None would cause the error because it does the current_task check before the None timeout check in that timeout class.

[thehesiod]

btw, can you try with the latest aiohttp? The codepath has changed significantly in regard to timeouts

[andymo]

Regarding Tornado's use of current_task, it was brought up in the two issues I linked previously, as well as most other issues opened regarding tornado + aio libraries. I figured since a fix did make its way into aiohttp, aiobotocore could support it. Otherwise I understand, it's not really aiobotocore's responsibility :)

I am on the latest pypi version of aiohttp (1.3.3), which should have all commits regarding timeouts that were pushed to fix aio-libs/aiohttp#1180, since I used this example and it was working. I see in the changelog of 1.3.4 that there is a revert of some timeout handling in client request, so I'll pull that and see if that helps, thanks!

[adamrothman]

@andymo The simplest fix is to just wrap your calls in a Task, like so:

async def get(self):
    session = aiobotocore.get_session()
    client = session.create_client('s3')

    task = asyncio.ensure_future(client.list_buckets())
    response = await task

    self.write(response)

[thehesiod]

btw, I'm disabling the use of aiohttp's read_timeout (#248) as it is implemented incorrectly, this may help your cause. If getting rid of conn timeout helps as well we can look into that. I'm really unhappy with their timeout impl :(

[jettify]

@thehesiod do you think we can close this?

[thehesiod]

I believe so