v4.24.9 breaks my ability to debug using delve

[trevorwhitney]

Describe the bug
We use this library in grafana/loki. After upgrading from v4.24.0-alpha.1 to v4.24.9, I lost my ability to use devle's DAP server on the Loki codebase. We've since downgraded to v4.24.8 and it works fine. I'm on Mac OS 15.0.1 (arm64), go 1.23.2, and delve 1.23.1. I wish I knew more about how delve worked to tell you what about that change broke it, but all I know is changing this specific dependency makes the difference.

To Reproduce
Use delve Dap before and after this version on a main.go that depends on it. I use nvim-dap and nvim-dap-go, so I'm not sure how to invoke the Dap manually, sorry I can't be more helpful. I'd be happy to try out any fixes in my environment to see if the problem persists.

Expected behavior
[A clear and concise description of what you expected to happen.]

Environment (please complete the following information):

• Windows: [paste the result of ver]
• Linux: [paste contents of /etc/os-release and the result of uname -a]
• Mac OS: [paste the result of sw_vers and uname -a

ProductName:		macOS
ProductVersion:		15.0.1
BuildVersion:		24A348

Darwin fiction.local 24.0.0 Darwin Kernel Version 24.0.0: Tue Sep 24 23:37:25 PDT 2024; root:xnu-11215.1.12~1/RELEASE_ARM64_T6030 arm64 arm Darwin

• FreeBSD: [paste the result of freebsd-version -k -r -u and uname -a]
• OpenBSD: [paste the result of uname -a]

[trevorwhitney]

I'm guessing this is related to #1702?

[uubulb]

Can you provide more details, like the content of the main.go or debug log?

[trevorwhitney]

@uubulb sorry for the delay, missed this in my email.

there's a lot of detail in go-delve/delve#3825, I'll do my best to summarize here.

here are the logs I was seeing:

[debug-adapter stderr] 2024-10-08T20:07:29-06:00 debug layer=dap DAP connection 1 started

[debug-adapter stderr] 2024-10-08T20:07:29-06:00 debug layer=dap [<- from client]{"seq":1,"type":"request","command":"initialize","arguments":{"clientID":"neovim","clientName":"neovim","adapterID":"nvim-dap","locale":"en_US.UTF-8","linesStartAt1":true,"columnsStartAt1":true,"pathFormat":"path","supportsVariableType":true,"supportsRunInTerminalRequest":true,"supportsProgressReporting":true,"supportsStartDebuggingRequest":true}}

[debug-adapter stderr] 2024-10-08T20:07:29-06:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":1,"success":true,"command":"initialize","body":{"supportsConfigurationDoneRequest":true,"supportsFunctionBreakpoints":true,"supportsConditionalBreakpoints":true,"supportsEvaluateForHovers":true,"supportsSetVariable":true,"supportsExceptionInfoRequest":true,"supportsDelayedStackTraceLoading":true,"supportsLogPoints":true,"supportsDisassembleRequest":true,"supportsClipboardContext":true,"supportsSteppingGranularity":true,"supportsInstructionBreakpoints":true}}

[debug-adapter stderr] 2024-10-08T20:07:29-06:00 debug layer=dap [<- from client]{"seq":2,"type":"request","command":"launch","arguments":{"program":"\/Users\/twhitney\/workspace\/loki\/cmd\/loki\/main.go","type":"go","args":["-config.file=\/Users\/twhitney\/workspace\/loki\/cmd\/loki\/loki-local-config.yaml"],"request":"launch","buildFlags":"","name":"Debug (Arguments)"}}

[debug-adapter stderr] 2024-10-08T20:07:29-06:00 debug layer=dap parsed launch config: {
	"mode": "debug",
	"program": "/Users/twhitney/workspace/loki/cmd/loki/main.go",
	"args": [
		"-config.file=/Users/twhitney/workspace/loki/cmd/loki/loki-local-config.yaml"
	],
	"buildFlags": {},
	"backend": "default",
	"stackTraceDepth": 50
}

[debug-adapter stderr] 2024-10-08T20:07:36-06:00 debug layer=dap building from "/Users/twhitney/workspace/loki": [go build -o /Users/twhitney/workspace/loki/__debug_bin2736654371 -gcflags all=-N -l /Users/twhitney/workspace/loki/cmd/loki/main.go]

[debug-adapter stderr] 2024-10-08T20:07:36-06:00 debug layer=dap launching binary '/Users/twhitney/workspace/loki/__debug_bin2736654371' with config: {
	"mode": "debug",
	"program": "/Users/twhitney/workspace/loki/cmd/loki/main.go",
	"args": [
		"-config.file=/Users/twhitney/workspace/loki/cmd/loki/loki-local-config.yaml"
	],
	"cwd": "/Users/twhitney/workspace/loki",
	"buildFlags": {},
	"output": "/Users/twhitney/workspace/loki/__debug_bin2736654371",
	"dlvCwd": "/Users/twhitney/workspace/loki",
	"backend": "default",
	"stackTraceDepth": 50
}
2024-10-08T20:07:36-06:00 info layer=debugger launching process with args: [/Users/twhitney/workspace/loki/__debug_bin2736654371 -config.file=/Users/twhitney/workspace/loki/cmd/loki/loki-local-config.yaml]

[debug-adapter stderr] 2024-10-08T20:07:37-06:00 debug layer=debugger entryPoint 0x104b28000 machoOff 0x100000000

[debug-adapter stderr] 2024-10-08T20:07:38-06:00 warning layer=debugger debug_frame workaround not applied: function internal/abi.(*RegArgs).Dump (at 0x104b29070) covered by 0x104b29070-0x104b29230

[debug-adapter stderr] 2024-10-08T20:07:38-06:00 debug layer=debugger Adding target 29771 "/Users/twhitney/workspace/loki/__debug_bin2736654371 -config.file=/Users/twhitney/workspace/loki/cmd/loki/loki-local-config.yaml"

[debug-adapter stderr] 2024-10-08T20:07:38-06:00 debug layer=dap [-> to client]{"seq":0,"type":"event","event":"initialized"}
2024-10-08T20:07:38-06:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":2,"success":true,"command":"launch"}

[debug-adapter stderr] 2024-10-08T20:07:38-06:00 debug layer=dap [<- from client]{"seq":3,"type":"request","command":"setBreakpoints","arguments":{"source":{"name":"main.go","path":"/Users/twhitney/workspace/loki/cmd/loki/main.go"},"breakpoints":[{"line":38}],"lines":[38]}}

[debug-adapter stderr] 2024-10-08T20:07:38-06:00 info layer=debugger created breakpoint: &api.Breakpoint{ID:1, Name:"sourceBp Path=\"/Users/twhitney/workspace/loki/cmd/loki/main.go\"  Line=38 Column=0", Addr:0x1082d9a38, Addrs:[]uint64{0x1082d9a38}, AddrPid:[]int{29771}, File:"/Users/twhitney/workspace/loki/cmd/loki/main.go", Line:38, FunctionName:"main.main", ExprString:"", Cond:"", HitCond:"", HitCondPerG:false, Tracepoint:false, TraceReturn:false, Goroutine:false, Stacktrace:0, Variables:[]string(nil), LoadArgs:(*api.LoadConfig)(nil), LoadLocals:(*api.LoadConfig)(nil), WatchExpr:"", WatchType:0x0, VerboseDescr:[]string(nil), HitCount:map[string]uint64{}, TotalHitCount:0x0, Disabled:false, UserData:interface {}(nil), RootFuncName:"", TraceFollowCalls:0}
2024-10-08T20:07:38-06:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":3,"success":true,"command":"setBreakpoints","body":{"breakpoints":[{"id":1,"verified":true,"source":{"name":"main.go","path":"/Users/twhitney/workspace/loki/cmd/loki/main.go"},"line":38}]}}

[debug-adapter stderr] 2024-10-08T20:07:38-06:00 debug layer=dap [<- from client]{"seq":4,"type":"request","command":"configurationDone"}
2024-10-08T20:07:38-06:00 debug layer=dap [-> to client]{"seq":0,"type":"event","event":"output","body":{"category":"console","output":"Type 'dlv help' for list of commands.\n"}}
2024-10-08T20:07:38-06:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":4,"success":true,"command":"configurationDone"}

[debug-adapter stderr] 2024-10-08T20:07:38-06:00 debug layer=debugger continuing
2024-10-08T20:07:38-06:00 debug layer=debugger ContinueOnce
Type 'dlv help' for list of commands.

[debug-adapter stderr] 2024-10-08T20:07:38-06:00 debug layer=dap [-> to client]{"seq":0,"type":"event","event":"terminated","body":{}}

[debug-adapter stderr] 2024-10-08T20:07:38-06:00 debug layer=dap DAP server stopping...
2024-10-08T20:07:38-06:00 debug layer=dap halting
2024-10-08T20:07:38-06:00 debug layer=dap process not running
2024-10-08T20:07:38-06:00 debug layer=dap [-> to client]{"seq":0,"type":"event","event":"output","body":{"category":"console","output":"Process 29771 has exited with status 11\n"}}
2024-10-08T20:07:38-06:00 debug layer=dap [-> to client]{"seq":0,"type":"event","event":"output","body":{"category":"console","output":"Detaching\n"}}
2024-10-08T20:07:38-06:00 debug layer=debugger detaching
2024-10-08T20:07:38-06:00 debug layer=dap DAP error: read tcp 127.0.0.1:53286->127.0.0.1:53288: use of closed network connection
2024-10-08T20:07:38-06:00 debug layer=dap DAP server stopped

here was the PR that fixed it for me: grafana/loki#14437 by downgrading this library

here is the main.go from before my commit to fix it by downgrading the dependency.

[uubulb]

@trevorwhitney how did you figure out that the problem is related to gopsutil? I noticed loki only use process in pkg/analytics/reporter.go. Does a simple program like this still cause your debugger to fail?

[trevorwhitney]

I went back in the Loki code to a time I remember the debugger working, and it did, so I did a git bisect to find the offending commit, which was just a dependency update commit from our CI automation. I went through that commit reverting 1 library at a time until the debugger worked again.

I'll try out your example in a bit and let you know.

[trevorwhitney]

interesting, I am able to run that simple program with the debugger. so I tried bumping gopsutil again in grafana/loki and I can run the debugger now. I have no idea what's changed between last week and today. closing, thanks for your help and kindness!

[trevorwhitney]

follow up, what's the purego behavior if CGO_ENABLED=0? That appears to be the key here.

Before this change, I could run the debugger with CGO_ENABLED=0. I have since removed that for other reasons, and that seems to be the culprit.

• gopsutil < 4.24.9 and CGO_ENABLED=0 can debug
• gopsutil >=4.24.9 and CGO_ENABLED=0 cannot debug
• unsetting CGO_ENABLED debugging works with either version

[Lomanic]

Are you able to debug this program with CGO_ENABLED=0 ?

follow up, what's the purego behavior if CGO_ENABLED=0? That appears to be the key here.

From a quick check it switches between their pure Go implementation and one using cgo, respectively https://github.com/ebitengine/purego/tree/450319b71c523c3716a29e3068ef333583744aea/internal/fakecgo and https://github.com/ebitengine/purego/tree/450319b71c523c3716a29e3068ef333583744aea/internal/cgo with runtime/cgo, from https://github.com/ebitengine/purego/blob/450319b71c523c3716a29e3068ef333583744aea/nocgo.go and https://github.com/ebitengine/purego/blob/450319b71c523c3716a29e3068ef333583744aea/cgo.go

Diving deeper than that will probably be way above my head.

This issue will probably have to be raised against the purego and/or delve developers I think, we are just (new) users of that library.