Restarting cluster_mode server loses connection

[davidbau]

Converting a pretty vanilla node.js service ("pencild") to use cluster_mode on pm2. Works fine when initially starting, but after doing a "pm2 restart pencild", it restarts the processes, but they no longer respond to the port.

Repro information: Runing on debian wheezy. Code on pm2 branch here: https://github.com/PencilCode/pencilcode-site/tree/pm2.

Config:
https://github.com/PencilCode/pencilcode-site/blob/pm2/processes.json

{
  "apps" : [{
    "name"             : "pencild",
    "script"           : "server/server.js",
    "instances"        : "2",
    "log_date_format"  : "YYYY-MM-DD HH:mm Z",
    "exec_interpreter" : "/usr/local/bin/node",
    "exec_mode"        : "cluster_mode",
    "env": {
        "NODE_ENV": "production",
        "PORT": "8818"
    },
    "port"             : 8818
  }]
}

Here is what happens when using it:

First run, this works great:

$ pm2 start test
.json 
Starting PM2 daemon...
Process launched
┌──────────┬────┬─────────┬──────┬────────┬───────────┬────────┬─────────────┬─────────────┐
│ App name │ id │ mode    │ PID  │ status │ restarted │ uptime │      memory │    watching │
├──────────┼────┼─────────┼──────┼────────┼───────────┼────────┼─────────────┼─────────────┤
│ pencild  │ 0  │ cluster │ 7532 │ online │         0 │ 0s     │ 19.559 MB   │ unactivated │
│ pencild  │ 1  │ cluster │ 7534 │ online │         0 │ 0s     │ 16.469 MB   │ unactivated │
└──────────┴────┴─────────┴──────┴────────┴───────────┴────────┴─────────────┴─────────────┘
 Use `pm2 desc[ribe] ` to get more details

$ lsof -i tcp:8818
COMMAND  PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
pm2:    7530 davidbau    0u  IPv4  27138      0t0  TCP *:8818 (LISTEN)
pm2:    7532 davidbau   13u  IPv4  27138      0t0  TCP *:8818 (LISTEN)
pm2:    7534 davidbau   13u  IPv4  27138      0t0  TCP *:8818 (LISTEN)

$ curl --proxy localhost:8818 http://pencilcode.net/load/
(outputs lots of stuff)

At this point, the server is listening on 8818 and working great. (Note that this server is sensitive to the Host: header, so we test it by using it as a proxy, so we can form requests with a "real" domain name.)

But then when restarting:

$ pm2 restart pencild
Restarting process by name pencild
[PM2] Process 0 restarted
[PM2] Process 1 restarted
┌──────────┬────┬─────────┬──────┬────────┬───────────┬────────┬─────────────┬─────────────┐
│ App name │ id │ mode    │ PID  │ status │ restarted │ uptime │      memory │    watching │
├──────────┼────┼─────────┼──────┼────────┼───────────┼────────┼─────────────┼─────────────┤
│ pencild  │ 0  │ cluster │ 7570 │ online │         1 │ 0s     │ 25.031 MB   │ unactivated │
│ pencild  │ 1  │ cluster │ 7576 │ online │         1 │ 0s     │ 25.031 MB   │ unactivated │
└──────────┴────┴─────────┴──────┴────────┴───────────┴────────┴─────────────┴─────────────┘
 Use `pm2 desc[ribe] ` to get more details
$ lsof -i tcp:8818
COMMAND  PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
pm2:    7530 davidbau    0u  IPv4  27138      0t0  TCP *:8818 (LISTEN)
pm2:    7570 davidbau    0u  IPv4  27138      0t0  TCP *:8818 (LISTEN)
pm2:    7570 davidbau   13u  IPv4  27138      0t0  TCP *:8818 (LISTEN)
pm2:    7576 davidbau    0u  IPv4  27138      0t0  TCP *:8818 (LISTEN)
pm2:    7576 davidbau   13u  IPv4  27138      0t0  TCP *:8818 (LISTEN)

$ curl --proxy localhost:8818 http://pencilcode.net/load/
(hangs)

At this point, connections to port 8818 hang. The server is unresponsive.

User error or bug?

[davidbau]

More details: I'm on node v0.10.31.

[soyuka]

Could you try using fork mode? Or with node v0.11.13, thanks.

[fernandoneto]

i think there's a compatibility problem beetwen pm2 and node v0.10.x in cluster mode, you must install node v0.11.x or use pm2 in fork-mode

[davidbau]

My app is incompatible with 11.x (not sure why yet), and I have to port it.
So it is taking a little time to run this test.

Are there workarounds for 10.x?

On Wed, Oct 1, 2014 at 7:04 AM, Fernando Neto [email protected]
wrote:

i think there's a compatibility problem with node v0.10.x nao cluster
mode, you have to install de node v0.11.x or use pm2 in cluster mode

—
Reply to this email directly or view it on GitHub
#717 (comment).

[soyuka]

{
  "apps" : [{
    "name"             : "pencild",
    "script"           : "server/server.js",
    "log_date_format"  : "YYYY-MM-DD HH:mm Z",
    "exec_mode"        : "fork_mode",
    "env": {
        "NODE_ENV": "production",
        "PORT": "8818"
    },
    "port"             : 8818
  }]
}

[davidbau]

Am I correct in assuming that fork_mode does not allow me to have multiple
processes serving the same port?

On Wed, Oct 1, 2014 at 9:23 AM, Antoine Bluchet [email protected]
wrote:

{
"apps" : [{
"name" : "pencild",
"script" : "server/server.js",
"log_date_format" : "YYYY-MM-DD HH:mm Z",
"exec_mode" : "fork_mode",
"env": {
"NODE_ENV": "production",
"PORT": "8818"
},
"port" : 8818
}]
}

—
Reply to this email directly or view it on GitHub
#717 (comment).

[fernandoneto]

@davidbau that's correct.
you can launch in fork_mod with option -i max pm2 will launch 2 process and the first run ok but the second will have a problem with the port. you will have a error like this

Error: listen EADDRINUSE

[soyuka]

Indeed but cluster is an experimental 0.10.3x feature: http://nodejs.org/api/cluster.html.

[fernandoneto]

#736

[Unitech]

Dont use cluster_mode with Node 0.10