Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime/cgocall: crash; unexpected return pc; running on macOS 15.1 Intel #70206

Closed
uhei opened this issue Nov 5, 2024 · 8 comments
Closed

runtime/cgocall: crash; unexpected return pc; running on macOS 15.1 Intel #70206

uhei opened this issue Nov 5, 2024 · 8 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Backlog

Comments

@uhei
Copy link

uhei commented Nov 5, 2024

Go version

go version go1.23.2 darwin/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/Users/uhei/Library/Caches/go-build'
GOENV='/Users/uhei/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/uhei/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/uhei/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/Cellar/go/1.23.2/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/local/Cellar/go/1.23.2/libexec/pkg/tool/darwin_amd64'
GOVCS=''
GOVERSION='go1.23.2'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/uhei/Library/Application Support/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/55/dpmktlgx1rnfzyry15jhb1dw0000gn/T/go-build667416701=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

Run falcodump on macOS 15.1 amd64 with two falco plugins (cloudtrail, gcpaudit).

What did you see happen?

runtime: g 17: unexpected return pc for runtime.cgocallback called from 0x2c6c0dd41
stack: frame={sp:0x1c00004ffb8, fp:0x1c00004ffe0} stack=[0x1c00004c000,0x1c000050000)
0x000001c00004feb8:  0x000000c000006798  0x000000c00015bf30 
0x000001c00004fec8:  0x020201030ea7d139  0x0000000000000000 
0x000001c00004fed8:  0x0000000200000003  0x000000030ec85320 <_cgoexp_7857655b0ec4_plugin_get_fields+0x0000000000000000> 
0x000001c00004fee8:  0x000000c000006700  0x000001c00004fee0 
0x000001c00004fef8:  0x000000030ea0e800 <runtime.cgocallbackg1.deferwrap2+0x0000000000000000>  0x000001c00004fecd 
0x000001c00004ff08:  0x00007ff7bff00000  0x0000000000000000 
0x000001c00004ff18:  0x000000c000006700  0x0000000000000000 
0x000001c00004ff28:  0x000001c00004fef8  0x000001c00004ff80 
0x000001c00004ff38:  0x000000030ea0e485 <runtime.cgocallbackg+0x0000000000000105>  0x000000030ec85320 <_cgoexp_7857655b0ec4_plugin_get_fields+0x0000000000000000> 
0x000001c00004ff48:  0x00007ff7bfefd2b0  0x0000000000000000 
0x000001c00004ff58:  0x00000002c6c0dd41  0x000001c00004ffe0 
0x000001c00004ff68:  0x0000000000000000  0x000000c000006700 
0x000001c00004ff78:  0x000000c00005e008  0x000001c00004ffa8 
0x000001c00004ff88:  0x000000030ea887e9 <runtime.cgocallbackg+0x0000000000000029>  0x000000030ec85320 <_cgoexp_7857655b0ec4_plugin_get_fields+0x0000000000000000> 
0x000001c00004ff98:  0x00007ff7bfefd2b0  0x0000000000000000 
0x000001c00004ffa8:  0x00007ff7bfefd248  0x000000030ea84bcc <runtime.cgocallback+0x00000000000000cc> 
0x000001c00004ffb8: <0x000000030ec85320 <_cgoexp_7857655b0ec4_plugin_get_fields+0x0000000000000000>  0x00007ff7bfefd2b0 
0x000001c00004ffc8:  0x0000000000000000  0x0000000000000000 
0x000001c00004ffd8:  0x00000002c6c0dd41 >0x0000000000000000 
0x000001c00004ffe8:  0x0000000000000000  0x0000000000000000 
0x000001c00004fff8:  0x0000000000000000 
fatal error: unknown caller pc

runtime stack:
runtime.throw({0x30f487751?, 0x0?})
	/usr/local/Cellar/go/1.23.2/libexec/src/runtime/panic.go:1067 +0x48 fp=0x7ff7bfefcf90 sp=0x7ff7bfefcf60 pc=0x30ea7c9c8
runtime.(*unwinder).next(0x7ff7bfefd038)
	/usr/local/Cellar/go/1.23.2/libexec/src/runtime/traceback.go:470 +0x27c fp=0x7ff7bfefd018 sp=0x7ff7bfefcf90 pc=0x30ea6943c
runtime.copystack(0xc000006700, 0x4000)
	/usr/local/Cellar/go/1.23.2/libexec/src/runtime/stack.go:944 +0x2fe fp=0x7ff7bfefd0f8 sp=0x7ff7bfefd018 pc=0x30ea5f35e
runtime.newstack()
	/usr/local/Cellar/go/1.23.2/libexec/src/runtime/stack.go:1126 +0x4c5 fp=0x7ff7bfefd230 sp=0x7ff7bfefd0f8 pc=0x30ea5f905
runtime.morestack()
	/usr/local/Cellar/go/1.23.2/libexec/src/runtime/asm_amd64.s:621 +0x7a fp=0x7ff7bfefd238 sp=0x7ff7bfefd230 pc=0x30ea82f3a

goroutine 17 gp=0xc000006700 m=1 mp=0xc00005e008 [copystack, locked to thread]:
slices.insertionSortCmpFunc[...]({0x1c00013fb08, 0x7, 0x8}, 0x0, 0x7, 0x30f6ea6e0)
	/usr/local/Cellar/go/1.23.2/libexec/src/slices/zsortanyfunc.go:10 +0x4d3 fp=0x1c00004e480 sp=0x1c00004e478 pc=0x30ec76213
slices.pdqsortCmpFunc[...]({0x1c00013fb08, 0x7, 0x8}, 0x0, 0x7, 0x3, 0x30f6ea6e0)
	/usr/local/Cellar/go/1.23.2/libexec/src/slices/zsortanyfunc.go:73 +0xdb fp=0x1c00004e770 sp=0x1c00004e480 pc=0x30ec70f9b
slices.SortFunc[...]({0x1c00013fb08, 0x7, 0x8}, 0x30f6ea6e0)
	/usr/local/Cellar/go/1.23.2/libexec/src/slices/sort.go:32 +0x87 fp=0x1c00004e7d8 sp=0x1c00004e770 pc=0x30ec70e67
encoding/json.typeFields({0x30f6ffd60, 0x30f6a6fc0})
	/usr/local/Cellar/go/1.23.2/libexec/src/encoding/json/encode.go:1186 +0x356 fp=0x1c00004f310 sp=0x1c00004e7d8 pc=0x30ec6f096
encoding/json.cachedTypeFields({0x30f6ffd60, 0x30f6a6fc0})
	/usr/local/Cellar/go/1.23.2/libexec/src/encoding/json/encode.go:1277 +0x1cb fp=0x1c00004f450 sp=0x1c00004f310 pc=0x30ec6476b
encoding/json.newStructEncoder({0x30f6ffd60, 0x30f6a6fc0})
	/usr/local/Cellar/go/1.23.2/libexec/src/encoding/json/encode.go:725 +0x48 fp=0x1c00004f508 sp=0x1c00004f450 pc=0x30ec614e8
encoding/json.newTypeEncoder({0x30f6ffd60, 0x30f6a6fc0}, 0x1)
	/usr/local/Cellar/go/1.23.2/libexec/src/encoding/json/encode.go:413 +0x49e fp=0x1c00004f580 sp=0x1c00004f508 pc=0x30ec5e91e
encoding/json.typeEncoder({0x30f6ffd60, 0x30f6a6fc0})
	/usr/local/Cellar/go/1.23.2/libexec/src/encoding/json/encode.go:366 +0x365 fp=0x1c00004f680 sp=0x1c00004f580 pc=0x30ec5e2a5
encoding/json.newArrayEncoder({0x30f6ffd60, 0x30f587b60})
	/usr/local/Cellar/go/1.23.2/libexec/src/encoding/json/encode.go:864 +0x46 fp=0x1c00004f6d0 sp=0x1c00004f680 pc=0x30ec631c6
encoding/json.newSliceEncoder({0x30f6ffd60, 0x30f587b60})
	/usr/local/Cellar/go/1.23.2/libexec/src/encoding/json/encode.go:843 +0x13c fp=0x1c00004f758 sp=0x1c00004f6d0 pc=0x30ec62f1c
encoding/json.newTypeEncoder({0x30f6ffd60, 0x30f587b60}, 0x1)
	/usr/local/Cellar/go/1.23.2/libexec/src/encoding/json/encode.go:417 +0x451 fp=0x1c00004f7d0 sp=0x1c00004f758 pc=0x30ec5e8d1
encoding/json.typeEncoder({0x30f6ffd60, 0x30f587b60})
	/usr/local/Cellar/go/1.23.2/libexec/src/encoding/json/encode.go:366 +0x365 fp=0x1c00004f8d0 sp=0x1c00004f7d0 pc=0x30ec5e2a5
encoding/json.newPtrEncoder({0x30f6ffd60, 0x30f587b20})
	/usr/local/Cellar/go/1.23.2/libexec/src/encoding/json/encode.go:892 +0x46 fp=0x1c00004f920 sp=0x1c00004f8d0 pc=0x30ec638a6
encoding/json.newTypeEncoder({0x30f6ffd60, 0x30f587b20}, 0x1)
	/usr/local/Cellar/go/1.23.2/libexec/src/encoding/json/encode.go:421 +0x41c fp=0x1c00004f998 sp=0x1c00004f920 pc=0x30ec5e89c
encoding/json.typeEncoder({0x30f6ffd60, 0x30f587b20})
	/usr/local/Cellar/go/1.23.2/libexec/src/encoding/json/encode.go:366 +0x365 fp=0x1c00004fa98 sp=0x1c00004f998 pc=0x30ec5e2a5
encoding/json.valueEncoder({0x30f587b20, 0x30fbb3000, 0x16})
	/usr/local/Cellar/go/1.23.2/libexec/src/encoding/json/encode.go:340 +0x6b fp=0x1c00004fae8 sp=0x1c00004fa98 pc=0x30ec5deeb
encoding/json.(*encodeState).reflectValue(0x1c0004c2600, {0x30f587b20, 0x30fbb3000, 0x16}, {0x0, 0x1})
	/usr/local/Cellar/go/1.23.2/libexec/src/encoding/json/encode.go:322 +0x45 fp=0x1c00004fb28 sp=0x1c00004fae8 pc=0x30ec5dde5
encoding/json.(*encodeState).marshal(0x1c0004c2600, {0x30f587b20, 0x30fbb3000}, {0x0, 0x1})
	/usr/local/Cellar/go/1.23.2/libexec/src/encoding/json/encode.go:298 +0x107 fp=0x1c00004fbd0 sp=0x1c00004fb28 pc=0x30ec5da27
encoding/json.Marshal({0x30f587b20, 0x30fbb3000})
	/usr/local/Cellar/go/1.23.2/libexec/src/encoding/json/encode.go:164 +0x175 fp=0x1c00004fd68 sp=0x1c00004fbd0 pc=0x30ec5ccb5
github.com/falcosecurity/plugin-sdk-go/pkg/sdk/symbols/fields.plugin_get_fields()
	/Users/uhei/go/pkg/mod/github.com/falcosecurity/[email protected]/pkg/sdk/symbols/fields/fields.go:52 +0x65 fp=0x1c00004fe50 sp=0x1c00004fd68 pc=0x30ec851e5
_cgoexp_7857655b0ec4_plugin_get_fields(0x7ff7bfefd2b0)
	_cgo_gotypes.go:44 +0x18 fp=0x1c00004fe88 sp=0x1c00004fe50 pc=0x30ec85338
runtime.cgocallbackg1(0x30ec85320, 0x7ff7bfefd2b0, 0x0)
	/usr/local/Cellar/go/1.23.2/libexec/src/runtime/cgocall.go:442 +0x234 fp=0x1c00004ff40 sp=0x1c00004fe88 pc=0x30ea0e754
runtime.cgocallbackg(0x30ec85320, 0x7ff7bfefd2b0, 0x0)
	/usr/local/Cellar/go/1.23.2/libexec/src/runtime/cgocall.go:361 +0x105 fp=0x1c00004ff90 sp=0x1c00004ff40 pc=0x30ea0e485
runtime.cgocallbackg(0x30ec85320, 0x7ff7bfefd2b0, 0x0)
	<autogenerated>:1 +0x29 fp=0x1c00004ffb8 sp=0x1c00004ff90 pc=0x30ea887e9
runtime.cgocallback(0x0, 0x0, 0x0)
	/usr/local/Cellar/go/1.23.2/libexec/src/runtime/asm_amd64.s:1084 +0xcc fp=0x1c00004ffe0 sp=0x1c00004ffb8 pc=0x30ea84bcc
runtime: g 17: unexpected return pc for runtime.cgocallback called from 0x2c6c0dd41
stack: frame={sp:0x1c00004ffb8, fp:0x1c00004ffe0} stack=[0x1c00004c000,0x1c000050000)
0x000001c00004feb8:  0x000000c000006798  0x000000c00015bf30 
0x000001c00004fec8:  0x020201030ea7d139  0x0000000000000000 
0x000001c00004fed8:  0x0000000200000003  0x000000030ec85320 <_cgoexp_7857655b0ec4_plugin_get_fields+0x0000000000000000> 
0x000001c00004fee8:  0x000000c000006700  0x000001c00004fee0 
0x000001c00004fef8:  0x000000030ea0e800 <runtime.cgocallbackg1.deferwrap2+0x0000000000000000>  0x000001c00004fecd 
0x000001c00004ff08:  0x00007ff7bff00000  0x0000000000000000 
0x000001c00004ff18:  0x000000c000006700  0x0000000000000000 
0x000001c00004ff28:  0x000001c00004fef8  0x000001c00004ff80 
0x000001c00004ff38:  0x000000030ea0e485 <runtime.cgocallbackg+0x0000000000000105>  0x000000030ec85320 <_cgoexp_7857655b0ec4_plugin_get_fields+0x0000000000000000> 
0x000001c00004ff48:  0x00007ff7bfefd2b0  0x0000000000000000 
0x000001c00004ff58:  0x00000002c6c0dd41  0x000001c00004ffe0 
0x000001c00004ff68:  0x0000000000000000  0x000000c000006700 
0x000001c00004ff78:  0x000000c00005e008  0x000001c00004ffa8 
0x000001c00004ff88:  0x000000030ea887e9 <runtime.cgocallbackg+0x0000000000000029>  0x000000030ec85320 <_cgoexp_7857655b0ec4_plugin_get_fields+0x0000000000000000> 
0x000001c00004ff98:  0x00007ff7bfefd2b0  0x0000000000000000 
0x000001c00004ffa8:  0x00007ff7bfefd248  0x000000030ea84bcc <runtime.cgocallback+0x00000000000000cc> 
0x000001c00004ffb8: <0x000000030ec85320 <_cgoexp_7857655b0ec4_plugin_get_fields+0x0000000000000000>  0x00007ff7bfefd2b0 
0x000001c00004ffc8:  0x0000000000000000  0x0000000000000000 
0x000001c00004ffd8:  0x00000002c6c0dd41 >0x0000000000000000 
0x000001c00004ffe8:  0x0000000000000000  0x0000000000000000 
0x000001c00004fff8:  0x0000000000000000

What did you expect to see?

falcodump runs without any error like it does on macOS 15.1 on arm (also with go 1.23.2 and the same plugins) or on Arch Linux x86.
@gabyhelp
Copy link

gabyhelp commented Nov 5, 2024

Related Issues and Documentation

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

@dr2chase dr2chase added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. compiler/runtime Issues related to the Go compiler and/or runtime. labels Nov 6, 2024
@mknyszek
Copy link
Contributor

mknyszek commented Nov 6, 2024

In triage, we're not exactly sure what's wrong here, but something definitely broke on the Go stack; does the C++ code involved here do anything weird?

  • How easily can you reproduce this?
  • Does the C++ code call into Go, and then back into C++? Or do you call from Go into C++?
  • Does it only happen with this version of Go?

It's also very weird that a failure like this only happens on Intel Macs.

@mknyszek mknyszek added this to the Backlog milestone Nov 6, 2024
@mknyszek mknyszek added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 6, 2024
@uhei
Copy link
Author

uhei commented Nov 6, 2024 •
edited
Loading

In triage, we're not exactly sure what's wrong here, but something definitely broke on the Go stack; does the C++ code involved here do anything weird?

Don't know as we only use the existing plugins. The code doesn't look weird to me.
The c++ code is here.

  • How easily can you reproduce this?

We can reproduce this reliable on two different Intel MacBooks.

  • Does the C++ code call into Go, and then back into C++? Or do you call from Go into C++?

As far as I can tell C++ code call into Go and then back into C++.

  • Does it only happen with this version of Go?

Have only tried 1.23.2 so far. Which version should I give it a try?

It's also very weird that a failure like this only happens on Intel Macs.

Yes. That's why I've opened this issue. Today we've run falcodump with the plugins successfully also on Windows x86. We have only this issue on macOS on Intel.

@ianlancetaylor
Copy link
Member

Is there a way that we can reproduce the problem ourselves?

@uhei
Copy link
Author

uhei commented Nov 7, 2024

Is there a way that we can reproduce the problem ourselves?

I'm trying to write a simple POC to reproduce this.

@cherrymui
Copy link
Member 

runtime: g 17: unexpected return pc for runtime.cgocallback called from 0x2c6c0dd41
stack: frame={sp:0x1c00004ffb8, fp:0x1c00004ffe0} stack=[0x1c00004c000,0x1c000050000)

This is at the very top of the stack. Usually there is a (fake) runtime.goexit frame there to mark the top of stack and stop the unwinding. But it's not there on this stack.

Perhaps there is some overflow on the nearby memory region (possibly another stack) that overwrites the top words of this stack?

@uhei
Copy link
Author

uhei commented Nov 8, 2024

After go clean -modcache and compiling/installing everything from scratch we're no longer able to reproduce this. What ever caused this, seems to be fixed.

Thanks for your support!

@uhei uhei closed this as completed Nov 8, 2024
@github-project-automation github-project-automation bot moved this from Todo to Done in Go Compiler / Runtime Nov 8, 2024
@ianlancetaylor
Copy link
Member

Thanks for following up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
Go Compiler / Runtime
Status: Done
Milestone
Backlog
Development

No branches or pull requests
6 participants
@mknyszek @dr2chase @uhei @ianlancetaylor @cherrymui @gabyhelp