Problems using custom DNS with aiohttp

[vgoklani]

I would like to use a custom DNS server with aiohttp and followed the example from here:

https://aiohttp.readthedocs.io/en/stable/client_advanced.html#resolving-using-custom-nameservers

import asyncio
import uvloop
import aiohttp
from aiohttp.resolver import AsyncResolver

asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
loop = asyncio.get_event_loop()

resolver = AsyncResolver(nameservers=["1.1.1.1", "1.0.0.1"])
session = aiohttp.ClientSession(loop=loop, connector=aiohttp.TCPConnector(limit=100, ssl=False, force_close=True,
                ttl_dns_cache=300, resolver=resolver), timeout=aiohttp.ClientTimeout(total=5))

async def f(url):
    response = await session.get(url)
    response.raise_for_status()
    return response

loop.run_until_complete(f("https://example.com"))

Unfortunately I don't understand the error message below:

OSErrorTraceback (most recent call last)
/opt/conda/lib/python3.6/site-packages/aiohttp/connector.py in _wrap_create_connection(self, req, timeout, client_error, *args, **kwargs)
    821             with CeilTimeout(timeout.sock_connect):
--> 822                 return await self._loop.create_connection(*args, **kwargs)
    823         except certificate_errors as exc:

/opt/conda/lib/python3.6/site-packages/uvloop/loop.pyx in create_connection()

/opt/conda/lib/python3.6/site-packages/uvloop/loop.pyx in uvloop.loop.Loop.create_connection()

/opt/conda/lib/python3.6/site-packages/uvloop/handles/tcp.pyx in uvloop.loop.TCPTransport.connect()

/opt/conda/lib/python3.6/site-packages/uvloop/handles/tcp.pyx in uvloop.loop._TCPConnectRequest.connect()

OSError: [Errno 99] Cannot assign requested address

The above exception was the direct cause of the following exception:

ClientConnectorErrorTraceback (most recent call last)
<ipython-input-1-622959d2ae35> in <module>()
     16     return response
     17
---> 18 loop.run_until_complete(f("https://example.com"))

/opt/conda/lib/python3.6/site-packages/uvloop/loop.pyx in uvloop.loop.Loop.run_until_complete()

<ipython-input-1-622959d2ae35> in f(url)
     12
     13 async def f(url):
---> 14     response = await session.get(url)
     15     response.raise_for_status()
     16     return response

/opt/conda/lib/python3.6/site-packages/aiohttp/client.py in _request(self, method, url, params, data, json, headers, skip_auto_headers, auth, allow_redirects, max_redirects, compress, chunked, expect100, read_until_eof, proxy, proxy_auth, timeout, verify_ssl, fingerprint, ssl_context, ssl, proxy_headers, trace_request_ctx)
    364                                 req,
    365                                 traces=traces,
--> 366                                 timeout=timeout
    367                             )
    368                     except asyncio.TimeoutError as exc:

/opt/conda/lib/python3.6/site-packages/aiohttp/connector.py in connect(self, req, traces, timeout)
    443
    444             try:
--> 445                 proto = await self._create_connection(req, traces, timeout)
    446                 if self._closed:
    447                     proto.close()

/opt/conda/lib/python3.6/site-packages/aiohttp/connector.py in _create_connection(self, req, traces, timeout)
    755         else:
    756             _, proto = await self._create_direct_connection(
--> 757                 req, traces, timeout)
    758
    759         return proto

/opt/conda/lib/python3.6/site-packages/aiohttp/connector.py in _create_direct_connection(self, req, traces, timeout, client_error)
    877             return transp, proto
    878         else:
--> 879             raise last_exc
    880
    881     async def _create_proxy_connection(self, req, traces, timeout):

/opt/conda/lib/python3.6/site-packages/aiohttp/connector.py in _create_direct_connection(self, req, traces, timeout, client_error)
    860                     server_hostname=hinfo['hostname'] if sslcontext else None,
    861                     local_addr=self._local_addr,
--> 862                     req=req, client_error=client_error)
    863             except ClientConnectorError as exc:
    864                 last_exc = exc

/opt/conda/lib/python3.6/site-packages/aiohttp/connector.py in _wrap_create_connection(self, req, timeout, client_error, *args, **kwargs)
    827             raise ClientConnectorSSLError(req.connection_key, exc) from exc
    828         except OSError as exc:
--> 829             raise client_error(req.connection_key, exc) from exc
    830
    831     async def _create_direct_connection(self, req, traces, timeout,

ClientConnectorError: Cannot connect to host example.com:443 ssl:None [Cannot assign requested address]

I received this error on both macOS 10.1.3.6 and Ubuntu 16.04.4 LTS

[asvetlov]

Looks like your DNS server is broken

[webknjaz]

@asvetlov those IPs in the snippet are addresses of new CloudFlare's public DNS servers (just like 8.8.8.8 is Google's)

[webknjaz]

However AFAIR they along with Mozilla are implementing something to cut off unrequested metadata, which is meant to protect users' privacy. (aka GDPR-ready)

[vgoklani]

Fixed here:

https://stackoverflow.com/questions/48007453/asynchronous-http-calls-using-aiohttp-asyncio-fail-with-cannot-connect-to-host/48008873#48008873

[webknjaz]

Looks like this still could've been handled by aiohttp better.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a [new issue] for related bugs.
If you feel like there's important points made in this discussion, please include those exceprts into that [new issue].
[new issue]: https://github.com/aio-libs/aiohttp/issues/new