Goerli crash on 1.8.0

[yrong]

Describe the bug

After running 8 hours, oom as following

Jun-06 10:54:06.062[]                 �[32minfo�[39m: Synced - slot: 5795070 - head: 0x54df…9b98 - exec-block: syncing(9132193 0xce54…) - finalized: 0x05b2…159a:181093 - peers: 39

<--- Last few GCs --->

[2717078:0x62ed4e0] 28144237 ms: Scavenge 1964.8 (2038.4) -> 1956.7 (2039.1) MB, 4.5 / 0.0 ms  (average mu = 0.356, current mu = 0.463) allocation failure; 
[2717078:0x62ed4e0] 28144290 ms: Scavenge 1969.8 (2039.6) -> 1960.2 (2041.9) MB, 4.1 / 0.0 ms  (average mu = 0.356, current mu = 0.463) allocation failure; 
[2717078:0x62ed4e0] 28144346 ms: Scavenge 1972.3 (2042.4) -> 1963.9 (2045.4) MB, 4.5 / 0.0 ms  (average mu = 0.356, current mu = 0.463) allocation failure; 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb7a940 node::Abort() [node]
 2: 0xa8e823  [node]
 3: 0xd5c940 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xd5cce7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xf3a3e5  [node]
 6: 0xf3b2e8 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xf4b7f3  [node]
 8: 0xf4c668 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xf4f835 v8::internal::Heap::HandleGCRequest() [node]
10: 0xecd98f v8::internal::StackGuard::HandleInterrupts() [node]
11: 0x12ce22f v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [node]
12: 0x16fb6b9  [node]
Aborted (core dumped)

Expected behavior

running without oom

Steps to reproduce

npx lodestar beacon --dataDir="$ethereum_data_dir" --network="goerli" --rest.namespace="*" --jwt-secret="./config/jwtsecret" --checkpointSyncUrl="https://sync-goerli.beaconcha.in" > "$output_dir/lodestar.log" 2>&1 &

Additional context

Similar issues could be referenced

#5349
#5549

Operating system

Linux

Lodestar version or commit hash

v1.8.0

[philknows]

Do you know how much memory you've allocated to nodeJS when running Lodestar? I think by default, it's set at 1400MB, which could potentially be why. Usually when running Lodestar, we set --max-old-space-size=4096 to the nodeJS environment to give it more than the default. 1.4GB is definitely too low.

[yrong]

Thanks a lot for the reply! Yes, did not explicitly configure before.

I would assume this value will be dynamic determined based on physically available in RAM. So do we have recommendation about minim hardware spec for running a goerli/mainnet node?

[nflaig]

I would assume this value will be dynamic determined based on physically available in RAM

It has to be explicitly set, --max-old-space-size is an option passed to V8 (runtime engine used by Nodejs) which specifies max size of old memory section, it is unused memory that the garbage collector needs to free up. If the garabage collector is not able to keep up and old memory goes above the limit, it will blow up which is what happened in your case.

Running Lodestar via docker or installing from source will set this value by default

lodestar/lodestar

Line 7 in c775ad7

node --trace-deprecation --max-old-space-size=4096 ./packages/cli/bin/lodestar.js "$@"

So do we have recommendation about minim hardware spec for running a goerli/mainnet node?

It depends on the network and how many validators you are running, mainnet right now requires bit more resources than goerli since the state is bigger.

CPU usage is quite low compared to execution client and in most cases memory usage will be around 4GB.

See Hardware specifications minimum / recommended for more details.