2.5.20 Uncaught Exception in update cycle (IPC closed)

[dstack]

To be clear, what I see in the logs is that the config update ran, and the process is actively restarting itself. This is followed by an error regarding the IPC stream being closed (I assume between the cli-updater process and the "gateway" process). This causes an uncaught exception, which in turn is being reported as a crash in our container (PCF on Azure).

Logs as follows:

2018-07-26T15:17:19.393-06:00 [APP/PROC/WEB/0] [OUT] Configuration change detected. Saving new config and Initiating reload
2018-07-26T15:17:19.399-06:00 [APP/PROC/WEB/0] [OUT] Recieved reload instruction. Proceeding to reload
2018-07-26T15:17:20.132-06:00 [APP/PROC/WEB/0] [OUT] Reload completed
2018-07-26T15:17:20.136-06:00 [APP/PROC/WEB/0] [ERR] Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
2018-07-26T15:17:20.136-06:00 [APP/PROC/WEB/0] [ERR] at ChildProcess.target.send (internal/child_process.js:627:16)
2018-07-26T15:17:20.136-06:00 [APP/PROC/WEB/0] [ERR] at Worker.send (internal/cluster/worker.js:40:28)
2018-07-26T15:17:20.136-06:00 [APP/PROC/WEB/0] [ERR] at disconnectAndShutdownWorker (/home/vcap/app/cli/lib/reload-cluster.js:150:16)
2018-07-26T15:17:20.136-06:00 [APP/PROC/WEB/0] [ERR] at EventEmitter.replaceAndTerminateWorker (/home/vcap/app/cli/lib/reload-cluster.js:124:5)
2018-07-26T15:17:20.136-06:00 [APP/PROC/WEB/0] [ERR] at EventEmitter.emit (events.js:182:13)
2018-07-26T15:17:20.136-06:00 [APP/PROC/WEB/0] [ERR] at emit (/home/vcap/app/cli/lib/reload-cluster.js:59:15)
2018-07-26T15:17:20.136-06:00 [APP/PROC/WEB/0] [ERR] at EventEmitter.emitWorkerDisconnect (/home/vcap/app/cli/lib/reload-cluster.js:171:5)
2018-07-26T15:17:20.136-06:00 [APP/PROC/WEB/0] [ERR] at ChildProcess.worker.process.once (internal/cluster/master.js:216:13)
2018-07-26T15:17:20.136-06:00 [APP/PROC/WEB/0] [ERR] at Object.onceWrapper (events.js:273:13)
2018-07-26T15:17:20.136-06:00 [APP/PROC/WEB/0] [ERR] at ChildProcess.emit (events.js:182:13)
2018-07-26T15:17:20.136-06:00 [APP/PROC/WEB/0] [ERR] at finish (internal/child_process.js:796:14)
2018-07-26T15:17:20.136-06:00 [APP/PROC/WEB/0] [ERR] at process._tickCallback (internal/process/next_tick.js:61:11)
2018-07-26T15:17:20.137-06:00 [APP/PROC/WEB/0] [OUT] Removing the socket file as part of cleanup
2018-07-26T15:17:20.158-06:00 [APP/PROC/WEB/0] [OUT] Exit status 0
2018-07-26T15:17:20.160-06:00 [CELL/SSHD/0] [OUT] Exit status 0
2018-07-26T15:17:20.329-06:00 [API/2] [OUT] Process has crashed with type: "web"
2018-07-26T15:17:20.343-06:00 [API/2] [OUT] App instance exited with guid f2eb4551-9e84-4b58-b3fe-139ddbab1b1e payload: {"instance"=>"e5481dbc-5386-4c86-6f92-6a70", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"Codependent step exited", "crash_count"=>1, "crash_timestamp"=>1532639840283117473, "version"=>"14919e7d-406f-4405-b261-cc50ffe3c515"}
2018-07-26T15:17:20.438-06:00 [CELL/0] [OUT] Cell 9450680d-5bba-47a0-8045-17c1ed51ca91 stopping instance e5481dbc-5386-4c86-6f92-6a70
2018-07-26T15:17:20.438-06:00 [CELL/0] [OUT] Cell 9450680d-5bba-47a0-8045-17c1ed51ca91 destroying container for instance e5481dbc-5386-4c86-6f92-6a70
2018-07-26T15:17:20.732-06:00 [CELL/0] [OUT] Cell 21f2e3b7-b8d2-4b68-bdde-25dac89e464f creating container for instance cc44a43f-e74c-4784-7bc8-7fcf
2018-07-26T15:17:21.464-06:00 [CELL/0] [OUT] Cell 9450680d-5bba-47a0-8045-17c1ed51ca91 successfully destroyed container for instance e5481dbc-5386-4c86-6f92-6a70
2018-07-26T15:17:21.782-06:00 [CELL/0] [OUT] Cell 21f2e3b7-b8d2-4b68-bdde-25dac89e464f successfully created container for instance cc44a43f-e74c-4784-7bc8-7fcf
2018-07-26T15:17:24.824-06:00 [CELL/0] [OUT] Starting health monitoring of container
2018-07-26T15:17:25.457-06:00 [APP/PROC/WEB/0] [OUT] > [email protected] start /home/vcap/app

From this point the container restarts and we go through the same cycle in 10 more minutes.

[srinandan]

@whitlockjc - FYI^

[KyleWiese]

Hi dstack,
There seems to be an incompatibility with the most recent node versions and edgemicro. Could you try changing the engines.node and engines.npm field in the package.json to be:

"engines": {
    "node": "8.11.2",
    "npm": "5.6.0"
  },

[dstack]

Thanks for the info. We got this implemented today (our buildpack situation is a bit of a mess I learned today). Over the past 30 minutes we have had 0 crashes due to this issue (expected 3). I am going to continue to monitor, but it looks like this is the solution.

[vilobhmm]

@dstack : I have couple of questions for you

1. What was the node and edgemicro version at the time of the crash ?
2. Was the issue resolved by downgrading the version as suggested by @KyleWiese ? Or were you able to get it working with the original node and edgemicro versions ?