Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Process.children() is slow #1183

Closed
pitrou opened this issue Nov 30, 2017 · 6 comments
Closed

Process.children() is slow #1183

pitrou opened this issue Nov 30, 2017 · 6 comments

Comments

@pitrou
Copy link
Contributor

pitrou commented Nov 30, 2017

>>> import psutil
>>> p = psutil.Process()
>>> p.children()
[]
>>> %timeit p.children()
36.1 ms ± 268 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

It seems half of that is calling process_iter():

>>> %timeit l = list(psutil.process_iter())
20.5 ms ± 232 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
@giampaolo
Copy link
Owner

giampaolo commented Nov 30, 2017

Hey Antoine! =)
Process.children() is inevitably slow because it has to obtain pid and ppid for all processes:

psutil/psutil/__init__.py

Lines 862 to 864 in 3a3598e

for p in process_iter():
try:
if p.ppid() == self.pid:

We can't get away from that.

With that said, it is true that we're doing more than just that because we also call create_time(), even more than once (here and here).

We can probably avoid that and define a utility function similar to what we have on windows:

psutil/psutil/__init__.py

Lines 851 to 854 in 3a3598e

if hasattr(_psplatform, 'ppid_map'):
# Windows only: obtain a {pid:ppid, ...} dict for all running
# processes in one shot (faster).
ppid_map = _psplatform.ppid_map()

...which just returns a {pid; ppid, ...} map for all processes in one shot.

@pitrou
Copy link
Contributor Author

pitrou commented Nov 30, 2017

Does that function have to be consistent? The problem is if the processes change while iterating on e.g. /proc.

@giampaolo
Copy link
Owner

A process which "changes" while iterating means it will disappear, in which case we'll just skip it.
They will also be skipped in case of AccessDenied (doc doesn't mention this - I will file a ticket for it).

@pitrou
Copy link
Contributor Author

pitrou commented Nov 30, 2017

I see, so we can just scan quickly over /proc?

@giampaolo
Copy link
Owner

Yes. Same goes for all other platforms except Windows which already does this in C.

pitrou added a commit to pitrou/psutil that referenced this issue Nov 30, 2017
This makes Process.children() 3x faster when there are no children.
giampaolo added a commit that referenced this issue Dec 1, 2017
* update HISTORY

* update doc

* #1183: speedup Process.children() by 2.2x

* fix windows err

* #1083 / #1084: implement linux-specific ppid_map() function speending things up from 2x to 2.4x

* add ESRCH to err handling

* update doc
@giampaolo
Copy link
Owner

Addresed in #1185 and #1186. Closing.

illwieckz added a commit to DaemonEngine/Urcheon that referenced this issue Apr 3, 2018
it can spawns more threads than cpu cores in some case, but that's OK
the idea is to spawn tasks as subprocesses within threads
so we count the threads of the subprocesses (and not our threads)

it can spawns more threads than cpu cores because an heavy task
can be spawned right after another heavy one was spawned but hasn't
found the time to spawn its core itself, is usually only happens
on very first tasks when nothing is already running

Process.children() is very slow, see:
giampaolo/psutil#1183

it's highly recommended to use python3-psutil >= 5.4.2
because some great speed-up were implemented
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants