Can't make 40k requests from client to server #1142
Comments
|
Hi, My Environment
|
|
Hi, |
|
async def run(loop, r):
url = "http://localhost:8080/{}"
tasks = []
# create instance of Semaphore
sem = asyncio.Semaphore(10)
async with ClientSession() as session
for i in range(r):
# pass Semaphore to every GET request
task = asyncio.ensure_future(bound_fetch(sem, url.format(i), session))
tasks.append(task)
...You can check and set file descriptors limit like so $ ulimit -n
$ sudo ulimit -n 10000 |
|
Also |
|
@eightnoteight here is the exceptions... |
|
2 questions then I guess...
|
I can confirm this as I have personally checked So there is small chance of not being cleaned up as both our environments are almost same. But can you(@ericfrederich) go to that directory and check how many files does it have? PS: I saw an |
|
@ericfrederich,
I guess it performs a little bit slower than requests to localhost and TIME_WAIT connections are being recycled |
|
@ericfrederich based on @popravich input and if your httpbin server takes more time to perform the requests than the aiohttp server, then one possible explanation is that TIME_WAIT connections expire after some time, so if local httpbin server takes more time then more ports are available and if aiohttp server takes less time to serve the requests then TIME_WAIT connections are not yet freed. |
|
I added this to the top of my client.py to capture a log of the number of open files every 0.1 seconds. cmd = 'gnome-terminal -e \'watch -n 0.1 "ls -al /proc/%d/fd | wc -l >> /tmp/n_files"\'' % os.getpid()
os.system(cmd)I'm at work now on a CentOS7 VM... same behavior. Also, with httpbin I was running it locally via gunicorn like their webpage instructs. |
Why aiohttp gives out the exceptionafter @popravich mentioned about Without a single client session the by using a single ClientSession from the client side https://github.com/KeepSafe/aiohttp/blob/master/aiohttp/connector.py#L114 the client will never create more than 20 connections, I suppose this is not the point of your experiment. So the reason that you are getting that exception is because you don't have any free ports anymore. Why gunicorn is working and aiohttp server not workingand about gunicorn working and aiohttp server not working, it is because gunicorn takes more time to serve requests, so some of those TIME_WAIT connections are freed by the OS when serving the nth request, but aiohttp is fast so before TIME_WAIT connections are freed by the OS the aiohttp performs nth request and fails. this is based on my observed readings on maximum number of TIME_WAIT connections in each test, with aiohttp server test system has more TIME_WAIT connections(19K) at some point, than with gunicorn test (12K) the amount of time a connection stays in TIME_WAIT state can be found in this file locally I used these commands to time them time python client.py 2>&1 > _.log
watch -n 0.1 'netstat -n | grep :8080 | wc' # this will give the number of ports which are still not freedsome articles on TIME_WAIT: |
|
Thanks for looking into this. I appreciate the answers. I know now not to create a new session for each request. My I'm not sure that gunicorn takes more time to serve requests. Watching the netstat I that under both circumstances they hit exactly 10k. but...When I changed them to make 20k requests the open connections for httpbin plateaued around 14k... with aiohttp it hit exactly 20k. They both took right around 32 seconds to complete. Maybe my understanding of this is wrong, but it seem http bin responds to requests just as fast (if not faster). Any idea why the number of open connections would plateau when going against a different server? Could the httpbin server clean these up better somehow?... though you said it isn't really the client or server but the OS? |
|
The difference between aiohttp server and gunicorn is that gunicorn |
|
This seems well explained now. Thanks for that! I have emailed the author of the blog asking him to update the code in it or at least reference this GitHub issue. Feel free to close it. |
|
Hello, I'm the author of blog post that started this discussion. Thanks for letting me know about the problem @ericfrederich . I updated my post and fixed usage of ClientSession see here: pawelmhm/pawelmhm.github.io@8346f25. After fixing session timings got much better. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Long story short
I cannot make 40k requests using a semaphore to limit concurrency.
This is based on code from here though actual code is in-lined below.
Expected behaviour
I should be able to make unlimited requests from an aiohttp client to an aiohttp server.
Actual behaviour
Exceptions are thrown. See below.
I was able to make 1 million requests to a locally running httpbin server running under gunicorn.
I dug into it a bit and wondered if ayncio it self didn't like having 40k tasks running, so I faked out the response by modifying
aiohttp/client.pywith the following code and makeClientSession.get()return thisFakeResponseContextManager. It worked just fine when I did this.So the problem isn't that I'm creating 40,000 or 1,000,000 tasks. Python's asyncio seems to handle that many tasks just fine. Something blows up with this aiohttp client in conjunction with the aiohttp server (again, because serving httpbin test server worked just fine too).
Steps to reproduce
server
client
Your environment
The text was updated successfully, but these errors were encountered: