Performance improvements for systems with many processes

The past few tags are related to performance improvements. Namely:

• Reduction of binary size (1.0.10)
• Improved systemd pattern processing (1.0.11)

Especially the improvement of systemd pattern processing speeds up the gathering. Here are some CPU profilings on systems with ~650 processes matched by systemd patterns:

Before:

# time /usr/sbin/psstat --cache-name testing --systemd :php*fpm* | wc -c
69515

real    0m3.749s
user    0m1.855s
sys    0m0.697s

# go tool pprof psstat cpu.pprof
(pprof) top -cum
Showing nodes accounting for 10ms, 1.85% of 540ms total
Showing top 10 nodes out of 151
      flat  flat%   sum%        cum   cum%
         0     0%     0%      270ms 50.00%  main.main
         0     0%     0%      270ms 50.00%  runtime.main
         0     0%     0%      200ms 37.04%  runtime.mcall
         0     0%     0%      200ms 37.04%  runtime.schedule
      10ms  1.85%  1.85%      170ms 31.48%  runtime.findrunnable
         0     0%  1.85%      160ms 29.63%  runtime.park_m
         0     0%  1.85%      120ms 22.22%  github.com/fuegas/psstat/process.PidsFromSystemdFlags
         0     0%  1.85%      110ms 20.37%  os/exec.(*Cmd).Output
         0     0%  1.85%      110ms 20.37%  os/exec.(*Cmd).Run
         0     0%  1.85%       90ms 16.67%  os/exec.(*Cmd).Start

After

# time /usr/sbin/psstat --cache-name testing --systemd :php*fpm* | wc -c
69513

real    0m1.275s
user    0m0.689s
sys    0m0.075s

# go tool pprof psstat cpu.pprof
(pprof) top -cum
Showing nodes accounting for 20ms, 13.33% of 150ms total
Showing top 10 nodes out of 86
      flat  flat%   sum%        cum   cum%
         0     0%     0%      110ms 73.33%  main.main
         0     0%     0%      110ms 73.33%  runtime.main
      20ms 13.33% 13.33%       50ms 33.33%  github.com/fuegas/psstat/process.(*Process).SumResources
         0     0% 13.33%       30ms 20.00%  github.com/fuegas/psstat/process.(*Process).ParseStat
         0     0% 13.33%       30ms 20.00%  github.com/fuegas/psstat/process.GatherAllProcs
         0     0% 13.33%       30ms 20.00%  github.com/fuegas/psstat/process.GatherAllProcsSingleThreaded
         0     0% 13.33%       30ms 20.00%  github.com/fuegas/psstat/process.NewProcess
         0     0% 13.33%       30ms 20.00%  runtime.mallocgc
         0     0% 13.33%       30ms 20.00%  runtime.mapiternext
         0     0% 13.33%       30ms 20.00%  runtime.systemstack

Single threaded gathering

On servers with many processes we see occasional load spikes due to the way psstat spawns threads and load is calculated. To mitigate these false positives of high load we've opted to default to single threaded gathering of proc information.

Use systemd id as name

If no name is specified using flags, use the Id of the systemd unit for reporting.

Limit number of threads to prevent high load notices

On systems with many cores/hyperthreading go spawns as many threads as it can. Due to the way load is calculated this can result in notices of high load as explained in https://blog.avast.com/investigation-of-regular-high-load-on-unused-machines-every-7-hours . To prevent these notices we limit the number of threads to max 4.