PM2 running all cluster processes 100% CPU

[angeloborrelli]

Hello,
I'm running ubuntu 15.04 - virtual machine with these spec:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 42
Model name: Intel Xeon E312xx (Sandy Bridge)
Stepping: 1
CPU MHz: 2394.442
BogoMIPS: 4788.88
Virtualization: VT-x
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 4096K

Node 5.9.0 + PM2 1.0.2 running 2 cluster app and in monitor i see all clusters at 100% CPU

htop shows no processes with high CPU load..only I noticed 9 PM God Daemon running

PM2 logs is empty (except for lines reporting apps successful startup).
Finally this is Ecosystem.json file that i use for launching apps by "pm2 start Ecosystem.json" :
{
"apps":[{
// Application #1
"name" : "server-http",
"script" : "server-http.js",
"watch" : true,
"merge_logs" : true,
"cwd" : "server",
"instances" : 0,
"exec_mode" : "cluster_mode",
"env": {
"PM2": true,
"NODE_ENV" : "production"
},
"env_production" : {
"NODE_ENV": "production"
},
"env_staging" : {
"NODE_ENV" : "staging",
"TEST" : true
}
},{
// Application #2
"name" : "server-socket",
"script" : "server-socket.js",
"node_args" : "--harmony --max-old-space-size=4096",
"watch" : true,
"merge_logs" : true,
"cwd" : "server",
"instances" : 0,
"exec_mode" : "cluster_mode",
"env": {
"PM2": true,
"NODE_ENV" : "production"
},
"env_production" : {
"NODE_ENV": "production"
},
"env_staging" : {
"NODE_ENV" : "staging",
"TEST" : true
}
}]
}

Everything was fine with node 5.7.0 and PM2 1.0.1
Please can you help?
Thank you very much.
Angelo

[rdubigny]

@angeloborrelli

I encounter a similar issue on AWS EC2 ubuntu 14.04 with node 5.7.1 an pm2 1.0.2 as root.

I have multiples PM2 v1.0.2: God Daemon processes running and they always take 100% of my 2 CPUs.

Any ideas ?

[rdubigny]

@angeloborrelli What if you remove the"watch" : true ? see http://pm2.keymetrics.io/docs/usage/watch-and-restart/ -> "If --watch is enabled, stopping it won’t stop watching:"

[angeloborrelli]

"watch:true" removed but no success...still got 100% CPU on all clusters. I suspect this could be related to node 5.9.0 and how pm2 check its processes CPU consumption. Maybe i should downgrade node or pm2 but how?

[angeloborrelli]

@rdubigny do u know how to downgrade pm2 and node?

[soyuka]

I suspect this could be related to node 5.9.0 and how pm2 check its processes CPU consumption

Just tried with node 5.9 and I don't see any different behavior on this side as cpu never rises more than 0.5% with pidusage alone (interval of 100 ms).

To downgrade node, it depends on how you installed node in the first place. With nvm it's easy as nvm alias default 5.6 && nvm use default. For pm2 you can sepcify the wanted semver when installing npm install [email protected].

[angeloborrelli]

@soyuka
successfully downgraded pm2 to ver 1.0.1 and "git hard reset" the project to a very old version but no way...pm2 monit is still showing 100% CPU
I used nvm to downgrade node to 5.7.1 but pm2 (both 101 + 102) could not start apps (hanging with "Error: spawn /usr/bin/nodejs ENOENT"). I guess that's why nvm installs node to its own directory but don't know how to solve this problem.

[angeloborrelli]

Restored PM2 1.0.2 and node 5.9.1 and made a simple test:

1. create a sample "app.js" like this:
   var http = require('http');

// Configure our HTTP server to respond with Hello World to all requests.
var server = http.createServer(function (request, response) {
response.writeHead(200, {"Content-Type": "text/plain"});
response.end("Hello World\n");
});

// Listen on port 8000, IP defaults to 127.0.0.1
server.listen(8000);

// Put a friendly message on the terminal
console.log("Server running at http://127.0.0.1:8000/");
2. started it up by "sudo pm2 start app.js"
3. this is what monitor first says:

then after 2/3 minutes

and after 4/5 minutes it goes 100%
I tried same configuration on another virtual machine and everything works fine.
Really don't know what's going on

[angeloborrelli]

Finally discover the problem:
https://bugs.launchpad.net/linux/+bug/1494350
Probably a live migration caused a vCPU steal time overflow. I just reboot and now all seems to work as expected.

[myflowpl]

angeloborrelli solution works for me
just upgraded and restarted the server
now it works perfect on:
VPS: Linux Debian-82-jessie-64-minimal 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux
node v4.4.3
pm2 1.1.2

previous version where i had on all processes 100% CPU was: Linux Debian-82-jessie-64-minimal 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04) x86_64 GNU/Linux

restarting the server was the crucial thing to do

[heijmerikx]

This took me quite some while to figure out and trying to investigate. But a reboot seemed to solve the case indeed.

[ghost]

use the command "pm2 update" and everything will be fine

[aaronlifton]

this is still a breaking issue