Crash during Garbage collection in TLSWrap

[surajwy]

• Version:v6.11.0
• Platform:Amazon Linux 4.9.27-14.31.amzn1.x86_64
• OpenSSL version: OpenSSL 1.0.1k-fips

I am experiencing a crash in PostGarbageCollectionProcessing call.
I do not have any reproducible case. But it happens 1-2 times a day in production.
Whenever I have got this crash, we were connecting to AWS's endpoint.

Below is my stack trace:

(lldb) v8 bt
 * SBThread: tid = 0x0000  * frame #0: 0x000000000111b346 node`node::TLSWrap::~TLSWrap() + 358
    frame #1: 0x000000000111b421 node`node::TLSWrap::~TLSWrap() + 17
    frame #2: 0x0000000000c85e5c node`v8::internal::GlobalHandles::DispatchPendingPhantomCallbacks(bool) + 252
    frame #3: 0x0000000000c860ba node`v8::internal::GlobalHandles::PostGarbageCollectionProcessing(v8::internal::GarbageCollector, v8::GCCallbackFlags) + 42
    frame #4: 0x0000000000ca6e37 node`v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) + 535
    frame #5: 0x0000000000ca771a node`v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, char const*, char const*, v8::GCCallbackFlags) + 330
    frame #6: 0x0000000000ca932e node`v8::internal::Heap::HandleGCRequest() + 222
    frame #7: 0x0000000000c55bac node`v8::internal::StackGuard::HandleInterrupts() + 796
    frame #8: 0x0000000000eb75fc node`v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) + 380
    frame #9: 0x00001ee4063079a7 <exit>

Contents of registers:

(lldb) register read
General Purpose Registers:
       rax = 0x0000000008617ac8
       rbx = 0x0000000008617950
       rcx = 0x1baffed00baffedf
       rdx = 0x0000000000000000
       rdi = 0x0000000000000000
       rsi = 0x0000000008617ac8
       rbp = 0x00007fff78159b10
       rsp = 0x00007fff78159b00
        r8 = 0x0000000000000001
        r9 = 0x00000000084fda90
       r10 = 0x0007312ef242a226
       r11 = 0x0000000000000246
       r12 = 0x0000000000000000
       r13 = 0x00000000000000ce
       r14 = 0x0000000000000001
       r15 = 0x0000000004dc9e80
       rip = 0x000000000111b346  node`node::TLSWrap::~TLSWrap() + 358
    rflags = 0x0000000000010246
        cs = 0x0000000000000033
        fs = 0x0000000000000000
        gs = 0x0000000000000000
        ss = 0x000000000000002b
        ds = 0x0000000000000000
        es = 0x0000000000000000

Disassembly of the top frame is attached here: disassembly.log

[bnoordhuis]

Looks like the instruction pointer ended up in an all zeros page. Two questions:

1. What node binary is this? What do node -p process.versions and node -p process.config print?
2. Can you reproduce with the latest v6.x, v6.12.0?

[surajwy]

1. I have downloaded this binary from nodejs.org.

node -p process.versions
{ http_parser: '2.7.0',
  node: '6.11.0',
  v8: '5.1.281.102',
  uv: '1.11.0',
  zlib: '1.2.11',
  ares: '1.10.1-DEV',
  icu: '58.2',
  modules: '48',
  openssl: '1.0.2k' }

node -p process.config
{ target_defaults: 
   { cflags: [],
     default_configuration: 'Release',
     defines: [],
     include_dirs: [],
     libraries: [] },
  variables: 
   { asan: 0,
     coverage: false,
     debug_devtools: 'node',
     force_dynamic_crt: 0,
     gas_version: '2.23',
     host_arch: 'x64',
     icu_data_file: 'icudt58l.dat',
     icu_data_in: '../../deps/icu-small/source/data/in/icudt58l.dat',
     icu_endianness: 'l',
     icu_gyp_path: 'tools/icu/icu-generic.gyp',
     icu_locales: 'en,root',
     icu_path: 'deps/icu-small',
     icu_small: true,
     icu_ver_major: '58',
     node_byteorder: 'little',
     node_enable_d8: false,
     node_enable_v8_vtunejit: false,
     node_install_npm: true,
     node_module_version: 48,
     node_no_browser_globals: false,
     node_prefix: '/',
     node_release_urlbase: 'https://nodejs.org/download/release/',
     node_shared: false,
     node_shared_cares: false,
     node_shared_http_parser: false,
     node_shared_libuv: false,
     node_shared_openssl: false,
     node_shared_zlib: false,
     node_tag: '',
     node_use_bundled_v8: true,
     node_use_dtrace: false,
     node_use_etw: false,
     node_use_lttng: false,
     node_use_openssl: true,
     node_use_perfctr: false,
     node_use_v8_platform: true,
     openssl_fips: '',
     openssl_no_asm: 0,
     shlib_suffix: 'so.48',
     target_arch: 'x64',
     uv_parent_path: '/deps/uv/',
     uv_use_dtrace: false,
     v8_enable_gdbjit: 0,
     v8_enable_i18n_support: 1,
     v8_inspector: true,
     v8_no_strict_aliasing: 1,
     v8_optimized_debug: 0,
     v8_random_seed: 0,
     v8_use_snapshot: true,
     want_separate_host_toolset: 0 } }

2. Will try to reproduce this on latest v6.x on Monday. Will share the results.
   Also, FYI, we have also tried this on node v8.6 and this crash was still happening.

[bnoordhuis]

Does your application contain sensitive information? If it doesn't, perhaps you can upload a core dump from the latest v6.x or v8.x? If you don't want to share the link publicly, my email is [email protected].

[surajwy]

Yes, my app contains sensitive information. Won't be able to share. Sorry about that.

We are trying to move our app from node v0.10.32 to an LTS node version. There are 2 more types of crashes and I will open separate git issues for them. We do have some add-on modules, but with these crash dumps, I am not able to pinpoint any issues with them.

[addaleax]

@surajwy Is there a chance you could try the patch from #17478?

[surajwy]

@addaleax, Yes, I will give it a try next week.

[apapirovski]

This got fixed in #17478 but the RegExp didn't pick it up due to the struck out words. @surajwy feel free to reopen if the issue persists after building from master with that patch. Thanks!