Cancerous memory consumption under specific circumstances #29001
Comments
This is a workaround for an issue that occurs in node 12.5.0 and newer, described in detail here: nodejs/node#29001 The behavior of the code in question is the same, however the change is enough to prevent locking the process into a "cancerous" state of memory growth. This is one of the weirdest bugs I've ever debugged. Under certain circumstances, Node process (starting from version 12.5.0, previous versions not affected), would not exit (even if we call `process.exit()` at the end of the file), and instead start to consume memory until it fills the whole available memory, using 100% CPU while at it. Here's a reproduction based on the `HasteMap`: https://gist.github.com/niieani/a527e9b2e6caf28e3021a50b938e4a91 Standard memory or CPU profiling doesn't show anything (literally, there are no function calls shown in the profiler). Dumping the heap snapshot results in a 36MB file, even when the memory consumption of the process is 10 GB. The backstory is that in our case [`jest`](https://github.com/facebook/jest) would not end AFTER all the tests have completed, and instead would start consuming huge amounts of memory (on CI, multiple processes from separate PRs would deplete the 128GB of RAM on the machine) so I started going down the rabbit hole to try and pinpoint what is the cause of the issue, which lead me to this minimal reproduction scenario, but also to discover all the quirks that helped to avoid triggering the scenario.
|
#28955 fixes this for me. |
|
Great to know we have a fix, but it would be useful to understand what was causing the issue in the first place, before closing. Otherwise we cannot know if the issue simply shifted around and isn't triggered by something slightly different (given how delicate reproducing this bug is). |
|
@niieani I don’t really have an answer for that – maybe somebody in @nodejs/v8 happens to know more. It’s a good sign that this is almost surely a bug in V8 and is fixed by a V8 upgrade; in the worst case, one could bisect V8 to figure out which commit exactly fixed this. |
|
For people looking at this from a "react-native start" is slow perspective, you can either use node <= v12.4.0, likely node >= 12.9.0 (assuming it gets the V8 sync #28955), or (new info) based on my testing just now you might try a workaround in jest-haste-map documented here with an attached patch-package compatible patch until jest-haste-map has the workaround integrated |
This is one of the weirdest bugs I've ever debugged.
Under certain circumstances, Node process (starting from version 12.5.0, previous versions not affected), will not exit (even if we call
process.exit()at the end of the file), and will start to consume memory until it fills the whole available memory, using 100% CPU while at it.Here's a reproduction:
https://gist.github.com/niieani/a527e9b2e6caf28e3021a50b938e4a91
Now, what's really funny, there are a few things that make this reproduction not trigger the bug:
reproduction.jsonshorter makes the process exit normallypath.resolvewith a call to a different function, e.g. byconsole.log(note that this originally usedjest-haste-map/build/lib/fast_pathand would still trigger the bug)Standard memory or CPU profiling doesn't show anything (literally, there are no function calls shown in the profiler). Dumping the heap snapshot results in a 36MB file, even when the memory consumption of the process is 10 GB.
The backstory is that in our case
jestwould not end AFTER all the tests have completed, and instead would start consuming huge amounts of memory (on CI, multiple processes from separate PRs would deplete the 128GB of RAM on the machine) so I started going down the rabbit hole to try and pinpoint what is the cause of the issue, which lead me to this minimal reproduction scenario.I've done a core dump, so here's what I got from
lldbfrom a slightly larger (earlier version) of my reproduction:core dump (lldb)
I couldn't get
llnodeto work, so this is all I could get from the above.The integrated report also wasn't very helpful, but here's the relevant part (report from an even earlier reproduction while running in
jest):Node Report
{ "javascriptStack": { "message": "No stack.", "stack": [ "Unavailable." ] }, "nativeStack": [ { "pc": "0x0000000100147402", "symbol": "report::TriggerNodeReport(v8::Isolate*, node::Environment*, char const*, char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, v8::Local<v8::String>) [/usr/local/bin/node]" }, { "pc": "0x000000010014df5c", "symbol": "report::WriteReport(v8::FunctionCallbackInfo<v8::Value> const&) [/usr/local/bin/node]" }, { "pc": "0x00000001001cf90c", "symbol": "v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [/usr/local/bin/node]" }, { "pc": "0x00000001001cefd6", "symbol": "v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/usr/local/bin/node]" }, { "pc": "0x00000001001ce7be", "symbol": "v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node]" }, { "pc": "0x000000010071ccd9", "symbol": "Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/usr/local/bin/node]" } ], "javascriptHeap": { "totalMemory": 43360256, "totalCommittedMemory": 41433760, "usedMemory": 38031208, "availableMemory": 2160634808, "memoryLimit": 2197815296, "heapSpaces": { "read_only_space": { "memorySize": 262144, "committedMemory": 32272, "capacity": 261832, "used": 31960, "available": 229872 }, "new_space": { "memorySize": 1048576, "committedMemory": 4520, "capacity": 1047328, "used": 3272, "available": 1044056 }, "old_space": { "memorySize": 35008512, "committedMemory": 34752088, "capacity": 34943712, "used": 32316584, "available": 2627128 }, "code_space": { "memorySize": 1212416, "committedMemory": 1072384, "capacity": 1150976, "used": 909120, "available": 241856 }, "map_space": { "memorySize": 2363392, "committedMemory": 2107280, "capacity": 2359128, "used": 1369600, "available": 989528 }, "large_object_space": { "memorySize": 3416064, "committedMemory": 3416064, "capacity": 3397216, "used": 3397216, "available": 0 }, "code_large_object_space": { "memorySize": 49152, "committedMemory": 49152, "capacity": 3456, "used": 3456, "available": 0 }, "new_large_object_space": { "memorySize": 0, "committedMemory": 0, "capacity": 1047328, "used": 0, "available": 1047328 } } }, "resourceUsage": { "userCpuSeconds": 17.6539, "kernelCpuSeconds": 1.12431, "cpuConsumptionPercent": 98.8324, "maxRss": 1796776919040, "pageFaults": { "IORequired": 2, "IONotRequired": 474408 }, "fsActivity": { "reads": 0, "writes": 0 } }, "libuv": [ { "type": "async", "is_active": true, "is_referenced": false, "address": "0x0000000104102f70" }, { "type": "timer", "is_active": true, "is_referenced": true, "address": "0x0000000105843328", "repeat": 0, "firesInMsFromNow": 616, "expired": false }, { "type": "check", "is_active": true, "is_referenced": false, "address": "0x00000001058433c0" }, { "type": "idle", "is_active": false, "is_referenced": true, "address": "0x0000000105843438" }, { "type": "prepare", "is_active": false, "is_referenced": false, "address": "0x00000001058434b0" }, { "type": "check", "is_active": false, "is_referenced": false, "address": "0x0000000105843528" }, { "type": "async", "is_active": true, "is_referenced": false, "address": "0x000000010422b840" }, { "type": "async", "is_active": true, "is_referenced": false, "address": "0x0000000101128958" }, { "type": "signal", "is_active": true, "is_referenced": false, "address": "0x0000000103f05970", "signum": 31, "signal": "SIGUSR2" }, { "type": "tty", "is_active": false, "is_referenced": true, "address": "0x0000000104231848", "width": 178, "height": 35, "fd": 20, "writeQueueSize": 0, "readable": true, "writable": true }, { "type": "signal", "is_active": true, "is_referenced": false, "address": "0x0000000104303240", "signum": 28, "signal": "SIGWINCH" }, { "type": "tty", "is_active": false, "is_referenced": true, "address": "0x0000000104305568", "width": 178, "height": 35, "fd": 22, "writeQueueSize": 0, "readable": true, "writable": true }, { "type": "loop", "is_active": true, "address": "0x000000010112e690" } ] }The text was updated successfully, but these errors were encountered: