Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VS Code renderer crashes when setting certain breakpoints #122225

Closed
rebornix opened this issue Apr 26, 2021 · 24 comments
Closed

VS Code renderer crashes when setting certain breakpoints #122225

rebornix opened this issue Apr 26, 2021 · 24 comments
Assignees
Labels
bug Issue identified by VS Code Team member as probable bug chromium Issues and items related to Chromium debug Debug viewlet, configurations, breakpoints, adapter issues insiders-released Patch has been released in VS Code Insiders papercut 🩸 A particularly annoying issue impacting someone on the team upstream Issue identified as 'upstream' component related (exists outside of VS Code) upstream-issue-fixed The underlying upstream issue has been fixed verified Verification succeeded

Comments

@rebornix
Copy link
Member

no 100% reproduce but very often that when I debug a unit test in vscode repo through the self host test extension, the process crashes.



#
# Fatal error in , line 0
# ignored
#
#
#
#FailureMessage Object: 0x7ffee59af6300   Electron Framework                  0x0000000112872879 v8::internal::SetupIsolateDelegate::SetupHeap(v8::internal::Heap*) + 6161385
1   Electron Framework                  0x0000000115965473 v8::internal::compiler::ZoneStats::GetCurrentAllocatedBytes() const + 12547347
2   Electron Framework                  0x00000001171f1f0d v8::internal::compiler::ZoneStats::GetCurrentAllocatedBytes() const + 38289325
3   Electron Framework                  0x00000001171dfad1 v8::internal::compiler::ZoneStats::GetCurrentAllocatedBytes() const + 38214513
4   Electron Framework                  0x0000000111f59012 v8::internal::RegExpParser::ParseRegExp(v8::internal::Isolate*, v8::internal::Zone*, v8::internal::FlatStringReader*, v8::base::Flags<v8::internal::JSRegExp::Flag, int>, v8::internal::RegExpCompileData*) + 1762
5   Electron Framework                  0x0000000111f53508 v8::internal::RegExpMacroAssembler::LoadCurrentCharacter(int, v8::internal::Label*, bool, int, int) + 1336
6   Electron Framework                  0x0000000111f5bb71 v8::internal::RegExp::Exec(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>, int, v8::internal::Handle<v8::internal::RegExpMatchInfo>) + 4465
7   Electron Framework                  0x0000000111f5aaf8 v8::internal::RegExp::Exec(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSRegExp>, v8::internal::Handle<v8::internal::String>, int, v8::internal::Handle<v8::internal::RegExpMatchInfo>) + 248
8   Electron Framework                  0x0000000111f6e0c1 v8::internal::Runtime::SetObjectProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::StoreOrigin, v8::Maybe<v8::internal::ShouldThrow>) + 19761
9   Electron Framework                  0x00000001123006f8 v8::internal::SetupIsolateDelegate::SetupHeap(v8::internal::Heap*) + 451176
10  Electron Framework                  0x0000000112351c7b v8::internal::SetupIsolateDelegate::SetupHeap(v8::internal::Heap*) + 784363
Renderer process crashed with: {"reason":"crashed","exitCode":4}
Waiting for the debugger to disconnect...

@rebornix rebornix added the papercut 🩸 A particularly annoying issue impacting someone on the team label Apr 26, 2021
@rebornix
Copy link
Member Author

rebornix commented Apr 26, 2021

Added paper cut label as it's kind of frustrating to debug unit test in our repo as it fails more than 90% of the time sometimes.

@rebornix
Copy link
Member Author

rebornix commented Apr 26, 2021

Sometimes the breakpoint just doesn't hit even if the debugger is attached.

Debugger listening on ws://127.0.0.1:56462/93767fef-f6bf-4c7f-958f-051cd7475bed
For help, see: https://nodejs.org/en/docs/inspector
Debugger attached.

DevTools listening on ws://127.0.0.1:9222/devtools/browser/1b2f568c-968a-4c46-91a4-e1ef7e6a811f

image

@connor4312
Copy link
Member

connor4312 commented Apr 26, 2021

#122225 (comment) will be fixed #121751. Currently the debugger attaching is entirely asynchronous to tests running, so this can happen. Might be fixed today if I get through my other misc remaining milestone work.

@deepak1556
Copy link
Collaborator

@connor4312
Copy link
Member

connor4312 commented Apr 26, 2021

I hit this on Windows just now,

Renderer process crashed with: {"reason":"crashed","exitCode":-1073741819}

Report: https://memes.peet.io/img/c3fce1de-6761-4bd9-a58c-12d3b59ac47e.dmp

It isn't the 100% repro that Peng has, so it may be different

@deepak1556
Copy link
Collaborator

The above is a crash due to this check https://source.chromium.org/chromium/chromium/src/+/master:v8/src/regexp/regexp-stack.cc;l=19. V8 recently added that check to prevent reentrant irregexp, not sure what triggered it in your case. Given the Regexp stack in Peng's post, very likely he is also hitting the same. Would be good to confirm with his crash report when available.

@weinand weinand added debug Debug viewlet, configurations, breakpoints, adapter issues bug Issue identified by VS Code Team member as probable bug labels Jul 5, 2021
@weinand
Copy link
Contributor

weinand commented Jul 5, 2021

@rebornix I'd like to reproduce this. Could you please provide detailed steps for

"when I debug a unit test in vscode repo through the self host test extension,"

@weinand weinand self-assigned this Jul 5, 2021
@rebornix
Copy link
Member Author

rebornix commented Jul 5, 2021

@weinand I wish I could provide stable reproduce steps. What happens right now on macOS when clicking on the run or debug action for a test

test('no rpc, createCommentController(...)', function () {
(remove this.skip)

  • When clicking on the icon on the gutter directly, it starts and finishes real quick but it doesn't fail (it should)
  • Add a breakpoint on the first line in the test, clicking on Debug test from context menu, breakpoint never hits and the only error I got is
[2021-07-05 14:15:01.047] [renderer2] [error] Cannot read property 'id' of undefined: TypeError: Cannot read property 'id' of undefined
    at t.ModuleTreeDataProvider.onChangedActiveDebugSession (/Users/penlv/.vscode-insiders/extensions/vadimcn.vscode-lldb-1.6.5/extension.js:1:35170)
    at d.fire (/Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/services/extensions/node/extensionHostProcess.js:57:1746)
    at w.$acceptDebugSessionActiveChanged (/Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/services/extensions/node/extensionHostProcess.js:89:34044)
    at u._doInvokeHandler (/Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/services/extensions/node/extensionHostProcess.js:88:12743)
    at u._invokeHandler (/Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/services/extensions/node/extensionHostProcess.js:88:12435)
    at u._receiveRequest (/Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/services/extensions/node/extensionHostProcess.js:88:11097)
    at u._receiveOneMessage (/Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/services/extensions/node/extensionHostProcess.js:88:10095)
    at /Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/services/extensions/node/extensionHostProcess.js:88:8004
    at d.fire (/Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/services/extensions/node/extensionHostProcess.js:57:1726)
    at h.fire (/Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/services/extensions/node/extensionHostProcess.js:65:15777)
    at /Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/services/extensions/node/extensionHostProcess.js:103:29793
    at d.fire (/Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/services/extensions/node/extensionHostProcess.js:57:1726)
    at h.fire (/Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/services/extensions/node/extensionHostProcess.js:65:15777)
    at r._receiveMessage (/Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/services/extensions/node/extensionHostProcess.js:65:21035)
    at /Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/services/extensions/node/extensionHostProcess.js:65:17921
    at d.fire (/Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/services/extensions/node/extensionHostProcess.js:57:1726)
    at c.acceptChunk (/Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/services/extensions/node/extensionHostProcess.js:65:13138)
    at /Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/services/extensions/node/extensionHostProcess.js:65:12486
    at Socket._ (/Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/workbench/services/extensions/node/extensionHostProcess.js:103:13107)
    at Socket.emit (events.js:315:20)
    at addChunk (internal/streams/readable.js:309:12)
    at readableAddChunk (internal/streams/readable.js:284:9)
    at Socket.Readable.push (internal/streams/readable.js:223:10)
    at Pipe.onStreamRead (internal/stream_base_commons.js:188:23)

@connor4312
Copy link
Member

That error looks like it's in some other misbehaving extension.

When clicking on the icon on the gutter directly, it starts and finishes real quick but it doesn't fail (it should)... Add a breakpoint on the first line in the test, clicking on Debug test from context menu, breakpoint never hits and the only error I got is

It sounds like the test just isn't getting run in that case, which may be unrelated to the crash issue. What's the output in "Show Test Output"? You can always check that if you suspect something is amiss.

@deepak1556
Copy link
Collaborator

deepak1556 commented Jul 15, 2021

@hediet @rebornix can you see if the crash happens with latest insiders that has updated chromium runtime, https://chromium-review.googlesource.com/c/v8/v8/+/2681950 very likely fixed the issue. Do run with https://github.com/microsoft/vscode/wiki/Native-Crash-Issues#creating-and-symbolicating-local-crash-reports so that we collect crash dumps when possible.

@deepak1556
Copy link
Collaborator

Got a new trace from @connor4312 that demonstrated the same crash, I will continue to add some logs in the runtime as originally planned.

@connor4312
Copy link
Member

This does seem to be much rarer now, however (from my subjective experience)

@weinand
Copy link
Contributor

weinand commented Jul 16, 2021

@hediet you've seen Connors' request? It would be great if you could help him with crash dumps.

@hediet
Copy link
Member

hediet commented Jul 17, 2021

Wouldn't it make sense to have crash reports enabled all the time when debugging Code OSS?

It is a little bit inconvenient that I have to touch launch.json to enable it - I always have to stash/unstash it when resolving merge conflicts etc.

@deepak1556
Copy link
Collaborator

Yeah it sounds good to have crash reporter enabled on OSS by default. I will have it log to .build/crashes which aligns with place we store the runtime executable.

As for the issue, I am still trying to get a reliable repro. But for now there is a workaround for the crash, you can add --js-flags="--regexp_interpret_all" to the runtime args in your debug config. This will use a slower path of the regex execution by avoiding the tier-up compiler that causes the crash.

@deepak1556 deepak1556 added the chromium Issues and items related to Chromium label Jul 23, 2021
@deepak1556 deepak1556 added this to the On Deck milestone Jul 26, 2021
@connor4312 connor4312 changed the title Crash on debugging unit test in VS Code VS Code renderer crashes when setting certain breakpoints Aug 23, 2021
@connor4312
Copy link
Member

Priority on this should be bumped since it also affects extension authors building web extensions -- working in the webworker extension host.

@deepak1556 deepak1556 modified the milestones: On Deck, September 2021 Aug 23, 2021
@connor4312
Copy link
Member

I've now included --js-flags="--regexp_interpret_all" in the launch config and selfhost test provider by default.

@weinand
Copy link
Contributor

weinand commented Aug 31, 2021

@connor4312 indeed, with that flag I'm no longer seeing the crashes!

@hediet
Copy link
Member

hediet commented Sep 10, 2021

I'm still seeing very frequent crashes :(

image

I'm using the default launch config without any modifications:

image

@deepak1556
Copy link
Collaborator

deepak1556 commented Sep 10, 2021

Update on what is known so far:

  1. Crash is triggered by CHECk at https://source.chromium.org/chromium/chromium/src/+/main:v8/src/regexp/regexp-stack.cc;l=19. It was added to avoid re-entrancy in Irregexp, basically no new regex should be executed on the same thread while there is an active Irregexp.

  2. Debugging VS Code with VS Code is flaky and frustrating #127861 (comment) provided a reliable way to repro the issue.

  3. Stacktrace https://gist.github.com/deepak1556/c1f197550930b33cae7fe4d7997699ee

  4. From 3) one of the CDP commands setBreakpointByUrl issued by vscode-js-debug uses regex. This CDP calls into v8 which eventually calls into native Regexp code https://source.chromium.org/chromium/chromium/src/+/main:v8/src/inspector/v8-regex.cc;l=49-105

  5. During one of the above regex calls an interrupt is issued, which continues to process the breakpoint CDP calls leading to reentrancy in the regexp stack.

With trace_regexp_bytecode v8 flag, I was able to track the regexp calls before the crash. Currently working on creating a minimal repro with the above CDP call.

@deepak1556 deepak1556 added upstream-issue-linked This is an upstream issue that has been reported upstream and removed upstream-issue-pending Issues that are caused by chromium but have not been reported due to pending minimal repro labels Sep 17, 2021
@deepak1556
Copy link
Collaborator

Crbug: https://bugs.chromium.org/p/chromium/issues/detail?id=1250646

@deepak1556 deepak1556 added upstream-issue-fixed The underlying upstream issue has been fixed and removed upstream-issue-linked This is an upstream issue that has been reported upstream labels Sep 23, 2021
@connor4312 connor4312 removed their assignment Sep 28, 2021
@lramos15 lramos15 added the verification-steps-needed Steps to verify are needed for verification label Oct 4, 2021
@lramos15
Copy link
Member

lramos15 commented Oct 4, 2021

@deepak1556 Is there a way to verify this? I've seen this crash a lot less frequently so I think it's been resolved but it seems from the issue there is no clear repro

@rebornix
Copy link
Member Author

rebornix commented Oct 4, 2021

I'm not able to see the crash anymore, though I'm currently seeing

C:\Users\rebor\code\vscode [main ≡]> ./scripts/test test/unit/electron/index.js --reporter full-json-stream --grep /^Notebook$/ --remote-debugging-port=9222 --js-flags="--regexp_interpret_all" --disable-features=CalculateNativeWinOcclusion --timeout=0 --waitServer=60901


DevTools listening on ws://127.0.0.1:9222/devtools/browser/ea612c4f-a167-4bbe-a44c-7525b482d69f
[31896:1004/140220.918:INFO:CONSOLE(29729)] "'window.webkitStorageInfo' is deprecated. Please use 'navigator.webkitTemporaryStorage' or 'navigator.webkitPersistentStorage' instead.", source: file:///C:/Users/rebor/code/vscode/node_modules/mocha/mocha.js (29729)
error connecting to waitServer Error: connect ECONNREFUSED 127.0.0.1:60901
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1146:16) {
  errno: -4078,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 60901
}
[31896:1004/140221.225:INFO:CONSOLE(33)] "The vm module of Node.js is deprecated in the renderer process and will be removed.", source: electron/js2c/renderer_init.js (33)
["start",{}]
["end",{"suites":0,"tests":0,"passes":0,"pending":0,"failures":0,"start":"2021-10-04T21:02:22.699Z","end":"2021-10-04T21:02:22.703Z","duration":4}]

@rzhao271
Copy link
Contributor

rzhao271 commented Oct 4, 2021

Marking as verified based on how I'm unable to repro #127861 (comment).

@connor4312 is the ECONNREFUSED issue above a separate issue?

@rzhao271 rzhao271 added verified Verification succeeded and removed verification-steps-needed Steps to verify are needed for verification labels Oct 4, 2021
@github-actions github-actions bot locked and limited conversation to collaborators Nov 15, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Issue identified by VS Code Team member as probable bug chromium Issues and items related to Chromium debug Debug viewlet, configurations, breakpoints, adapter issues insiders-released Patch has been released in VS Code Insiders papercut 🩸 A particularly annoying issue impacting someone on the team upstream Issue identified as 'upstream' component related (exists outside of VS Code) upstream-issue-fixed The underlying upstream issue has been fixed verified Verification succeeded
Projects
None yet
Development

No branches or pull requests

8 participants
@rebornix @deepak1556 @weinand @connor4312 @hediet @lramos15 @rzhao271 and others