Extension host crashes every time I open this large file

[eamodio]

Issue Type: Bug

extension-prod.zip

This happens even with starting vscode with all extensions disabled.

Each time I open this file in vscode the extension host will crash after a bit and this will be in the console:

Extension Host
extensionHost.ts:202 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 00007FF7F860C9E2 node::EmitAsyncDestroy+11839202
 2: 00007FF7F7A99A96 node::Abort+22
 3: 00007FF7F7A9A073 node::Abort+1523
 4: 00007FF7F560CCC8 v8::V8::GetVersion+792
 5: 00007FF7F560CC6A v8::V8::GetVersion+698
 6: 00007FF7F5655F83 std::_Vector_alloc<std::_Vec_base_types<v8::CpuProfileDeoptInfo,std::allocator<v8::CpuProfileDeoptInfo> > >::_Make_iterator+61731
 7: 00007FF7F5657653 std::_Vector_alloc<std::_Vec_base_types<v8::CpuProfileDeoptInfo,std::allocator<v8::CpuProfileDeoptInfo> > >::_Make_iterator+67571
 8: 00007FF7F5654377 std::_Vector_alloc<std::_Vec_base_types<v8::CpuProfileDeoptInfo,std::allocator<v8::CpuProfileDeoptInfo> > >::_Make_iterator+54551
 9: 00007FF7F565226A std::_Vector_alloc<std::_Vec_base_types<v8::CpuProfileDeoptInfo,std::allocator<v8::CpuProfileDeoptInfo> > >::_Make_iterator+46090
10: 00007FF7F565CB14 std::_Vector_alloc<std::_Vec_base_types<v8::CpuProfileDeoptInfo,std::allocator<v8::CpuProfileDeoptInfo> > >::_Make_iterator+89268
11: 00007FF7F565D03E std::_Vector_alloc<std::_Vec_base_types<v8::CpuProfileDeoptInfo,std::allocator<v8::CpuProfileDeoptInfo> > >::_Make_iterator+90590
12: 00007FF7F583C009 v8::Unlocker::~Unlocker+85785
13: 00007FF7F59B5E09 v8::Platform::SystemClockTimeMillis+969033
14: 00007FF7F599782E v8::Platform::SystemClockTimeMillis+844654
15: 00007FF7F59B3D16 v8::Platform::SystemClockTimeMillis+960598
16: 00007FF7F59B3BA0 v8::Platform::SystemClockTimeMillis+960224
17: 00007FF7F59B3858 v8::Platform::SystemClockTimeMillis+959384
18: 00007FF7F59AB7B6 v8::Platform::SystemClockTimeMillis+926454
19: 00007FF7F59AB390 v8::Platform::SystemClockTimeMillis+925392
20: 00007FF7F589CF3A v8::Unlocker::~Unlocker+482890
21: 00007FF7F589DD2F v8::Unlocker::~Unlocker+486463
22: 00007FF7F589DBBD v8::Unlocker::~Unlocker+486093
23: 00007FF7F58A321C v8::Unlocker::~Unlocker+508204
24: 00007FF7F5FD90FD v8_inspector::V8ContextInfo::executionContextId+3107101
25: 0000014BF4E2160C 
abstractExtensionService.ts:155 Extension host terminated unexpectedly. Code:  134  Signal:  null
_onExtensionHostCrashed @ abstractExtensionService.ts:155
_onExtensionHostCrashed @ extensionService.ts:375
_onExtensionHostCrashOrExit @ abstractExtensionService.ts:147
(anonymous) @ abstractExtensionService.ts:137
fire @ event.ts:580
_onExtHostProcessExit @ extensionHost.ts:467
(anonymous) @ extensionHost.ts:216
emit @ events.js:200
ChildProcess._handle.onexit @ internal/child_process.js:272
notificationsAlerts.ts:40 Extension host terminated unexpectedly.

Related to: #84939

VS Code version: Code - Insiders 1.41.0-insider (bf7d03b, 2019-11-15T05:40:12.201Z)
OS version: Windows_NT x64 10.0.19023

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz (8 x 4008)
GPU Status	2d_canvas: enabled flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled metal: disabled_off multiple_raster_threads: enabled_on oop_rasterization: disabled_off protected_video_decode: unavailable_off rasterization: enabled skia_renderer: disabled_off surface_control: disabled_off surface_synchronization: enabled_on video_decode: enabled viz_display_compositor: enabled_on viz_hit_test_surface_layer: disabled_off webgl: enabled webgl2: enabled
Load (avg)	undefined
Memory (System)	31.93GB (18.14GB free)
Process Argv	--enable-proposed-api eamodio.remotehub
Screen Reader	no
VM	0%

[alexdima]

I have also tried on Windows, but I cannot reproduce the OOM crash. I have instead took a heap snapshot and noticed that we cache TextDocument.lineAt calls, and I have removed that.

After removing that big memory hog, the largest objects retained come from the TS extension:

In blue, there is a memory leak which ends up referencing openFiles from nulToken. I believe it is not a good idea to reuse the same cancellation token everywhere, as if a listener is not correctly disposed, then it will leak and continue to be referenced by the nulToken. I suggest to simply allocate a new cancellation token for each call and remove nulToken.

In red, there is some sort of cache for the TypeScriptDocumentSymbolProvider that ends up consuming MB and MB...

[alexdima]

FYI @mjbvz I have pushed cf624b0 which removes the memory leak around nulToken, making its implementation similar to the implementation for CancellationToken.None in our core. i.e. with an event which never fires and which does not keep track of its listeners.

[alexdima]

I have disabled the caching done by CachedResponse to see what else shows up. I'll leave this to @mjbvz to decide what to do there, that cache should maybe expire after some time?...

The next kind of large memory leak appears to be in ProtocolBuffer in wireProtocol.ts:

[alexdima]

@dbaeumer maybe the same code pattern of wireProtocol.ts is used in other places. The ProtocolBuffer has a leak where its internal buffer always grows, so if a response has the size 15MB, the internal buffer will always remain at 15MB and never shrink to something reasonable again.

[alexdima]

FWIW I had the exact same problem and wrote ChunkStream here --

vscode/src/vs/base/parts/ipc/common/ipc.net.ts

Lines 29 to 120 in 515a0cb

	export class ChunkStream {
	private _chunks: VSBuffer[];
	private _totalLength: number;
	public get byteLength() {
	return this._totalLength;
	}
	constructor() {
	this._chunks = [];
	this._totalLength = 0;
	}
	public acceptChunk(buff: VSBuffer) {
	this._chunks.push(buff);
	this._totalLength += buff.byteLength;
	}
	public read(byteCount: number): VSBuffer {
	return this._read(byteCount, true);
	}
	public peek(byteCount: number): VSBuffer {
	return this._read(byteCount, false);
	}
	private _read(byteCount: number, advance: boolean): VSBuffer {
	if (byteCount === 0) {
	return getEmptyBuffer();
	}
	if (byteCount > this._totalLength) {
	throw new Error(`Cannot read so many bytes!`);
	}
	if (this._chunks[0].byteLength === byteCount) {
	// super fast path, precisely first chunk must be returned
	const result = this._chunks[0];
	if (advance) {
	this._chunks.shift();
	this._totalLength -= byteCount;
	}
	return result;
	}
	if (this._chunks[0].byteLength > byteCount) {
	// fast path, the reading is entirely within the first chunk
	const result = this._chunks[0].slice(0, byteCount);
	if (advance) {
	this._chunks[0] = this._chunks[0].slice(byteCount);
	this._totalLength -= byteCount;
	}
	return result;
	}
	let result = VSBuffer.alloc(byteCount);
	let resultOffset = 0;
	let chunkIndex = 0;
	while (byteCount > 0) {
	const chunk = this._chunks[chunkIndex];
	if (chunk.byteLength > byteCount) {
	// this chunk will survive
	const chunkPart = chunk.slice(0, byteCount);
	result.set(chunkPart, resultOffset);
	resultOffset += byteCount;
	if (advance) {
	this._chunks[chunkIndex] = chunk.slice(byteCount);
	this._totalLength -= byteCount;
	}
	byteCount -= byteCount;
	} else {
	// this chunk will be entirely read
	result.set(chunk, resultOffset);
	resultOffset += chunk.byteLength;
	if (advance) {
	this._chunks.shift();
	this._totalLength -= chunk.byteLength;
	} else {
	chunkIndex++;
	}
	byteCount -= chunk.byteLength;
	}
	}
	return result;
	}
	}

[dbaeumer]

@alexandrudima thanks for pinging. ActuallyI do: https://github.com/Microsoft/vscode-languageserver-node/blob/dbaeumer/callHierarchy2/jsonrpc/src/messageReader.ts#L42

Created microsoft/vscode-languageserver-node#544

[mjbvz]

Thanks for the investigation and fix @alexandrudima.

I've opened #85565 to track addressing the the buffer usage issue. I can no longer repo the original crash so I'm closing this issue for verification

[eamodio]

I can still reproduce the issue with that same file -- it takes about 30+ seconds for the EH to crash.

[mjbvz]

Marking as windows only since it doesn't happen for me on macos. @eamodio Does this happen with --disable-extension too?

[eamodio]

Yes

[eamodio]

So I've dug into this more and found that it is the "javascript.implicitProjectConfig.checkJs": true setting that when enabled causes the extension host to crash -- with it off, no crash.

[mjbvz]

@eamodio If you are still interesting in investigating this, I suspect it's not that setting itself that causes problems but the fact that it causes a very large number of diagnostics to be generated

[ronenl]

happens to me also in my mac

[mjbvz]

I looked into this this iteration but still can't repo a crash it on MacOS. There are definitely some slow downs however, specifically it seems around document symbols (partially because we keep reallocating the buffer when first getting the massive response json back)

[mjbvz]

Closing since we have not seen many other reports of this issue