getting timeout after streaming logs for some time

[TomasTomecek]

This is very likely an engine issue, first I would like to open here.

It looks like there is a change in behavior. When streaming logs with 1.10 I get timeout:

Traceback (most recent call last):
  File "/usr/lib64/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/home/tt/g/sen/sen/tui/widgets/list/common.py", line 43, in fetch_logs
    for line in generator:
  File "/home/tt/.local/lib/python3.5/site-packages/docker/client.py", line 279, in _stream_raw_result
    for out in response.iter_content(chunk_size=1, decode_unicode=True):
  File "/usr/lib/python3.5/site-packages/requests/utils.py", line 363, in stream_decode_response_unicode
    for chunk in iterator:
  File "/usr/lib/python3.5/site-packages/requests/models.py", line 667, in generate
    raise ConnectionError(e)
requests.exceptions.ConnectionError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.

generator is client.logs(c, stream=True)

This repro should be sufficient: #919 (comment)

I know this also happens when using compose.

[shin-]

Hmm, I tested against 1.9.0 and the same thing happens. What's your version of requests?

[TomasTomecek]

$ pip3 freeze 2>/dev/null | egrep "(requests=|urllib)"
requests==2.9.1
urllib3==1.13.1

docker-py is git master (c3a66cc)

Client:
 Version:      1.10.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   590d5108
 Built:        Thu Feb  4 19:55:25 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   590d5108
 Built:        Thu Feb  4 19:55:25 2016
 OS/Arch:      linux/amd64

Whole stack trace

Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/requests/packages/urllib3/response.py", line 226, in _error_catcher
    yield
  File "/usr/lib/python3.5/site-packages/requests/packages/urllib3/response.py", line 486, in read_chunked
    self._update_chunk_length()
  File "/usr/lib/python3.5/site-packages/requests/packages/urllib3/response.py", line 432, in _update_chunk_length
    line = self._fp.fp.readline()
  File "/usr/lib64/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/requests/models.py", line 660, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/usr/lib/python3.5/site-packages/requests/packages/urllib3/response.py", line 340, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/usr/lib/python3.5/site-packages/requests/packages/urllib3/response.py", line 514, in read_chunked
    self._original_response.close()
  File "/usr/lib64/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3.5/site-packages/requests/packages/urllib3/response.py", line 231, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
requests.packages.urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./test.py", line 10, in <module>
    for line in d.logs(container=restore, stream=True):
  File "/home/tt/g/docker-py/docker/client.py", line 279, in _stream_raw_result
    for out in response.iter_content(chunk_size=1, decode_unicode=True):
  File "/usr/lib/python3.5/site-packages/requests/utils.py", line 363, in stream_decode_response_unicode
    for chunk in iterator:
  File "/usr/lib/python3.5/site-packages/requests/models.py", line 667, in generate
    raise ConnectionError(e)
requests.exceptions.ConnectionError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.

[jasonamyers]

I have this same issue with 1.4.0 of docker-py and 1.9.1 of docker-engine. It hits us when we try to push a large (>4gb) container.

[christophlsa]

I had the same problem (requests = 2.3.0). Then I created a new virtual environment and installed the package(s) with the newest versions (requests = 2.9.1) and I had no problems anymore.

[ismaelfernandezscmspain]

I have the same problem, any answer about this?
I have installed (requests = 2.9.1) and (docker-py =1.7.2)

[akki]

I hit this (read timeout) error today while using the service_logs api.

Python 3.6, requests==2.22.0 FWIW

[zultron]

On a detached container, for l in container.logs(stream=True, follow=True): print(l) times out after a while with the following traceback:

  File "/usr/local/lib/python2.7/dist-packages/docker/types/daemon.py", line 32, in __next__
    return next(self._stream)
  File "/usr/local/lib/python2.7/dist-packages/docker/api/client.py", line 396, in _stream_raw_result
    for out in response.iter_content(chunk_size, decode):
  File "/usr/local/lib/python2.7/dist-packages/requests/utils.py", line 500, in stream_decode_response_unicode
    for item in iterator:
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 757, in generate
    raise ConnectionError(e)
requests.exceptions.ConnectionError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.

This happens when the container has no console output for long enough to exceed the timeout. A couple of workarounds helped:

• Disable the timeout with client = docker.from_env(timeout=None) (None means no timeout to urllib3)
• Set up a 'heartbeat' to periodically print to the console

This sounds like the issue @TomasTomecek. It also makes sense that @jasonamyers and @akki might hit this with other parts of the API that can possibly wait a long time for the server to respond.

While I'm not familiar with the code, ideally, these methods would set an infinite timeout, while other methods would set a reasonable default timeout.

[segevfiner]

I think I made a PR for this.

[zultron]

Indeed you did; I missed the pingback from #1959.

That patch works for me in place of the workarounds in my previous comment.

+1