Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fatal error: concurrent map iteration and map write on adsz debug endpoint #28958

Closed
howardjohn opened this issue Nov 17, 2020 · 0 comments · Fixed by #28968
Closed

fatal error: concurrent map iteration and map write on adsz debug endpoint #28958

howardjohn opened this issue Nov 17, 2020 · 0 comments · Fixed by #28968
Labels
area/networking

Comments

@howardjohn
Copy link
Member 

fatal error: concurrent map iteration and map write

goroutine 82039 [running]:
runtime.throw(0x2d936c2, 0x26)
        /usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc0035178f8 sp=0xc0035178c8 pc=0x436912
runtime.mapiternext(0xc003517ac0)
        /usr/local/go/src/runtime/map.go:853 +0x552 fp=0xc003517978 sp=0xc0035178f8 pc=0x411522
istio.io/istio/pilot/pkg/xds.(*DiscoveryServer).adsz(0xc0002ee6c0, 0x324a740, 0xc002d45500, 0xc009d5bc00)
        /home/howardjohn/go/src/istio.io/istio/pilot/pkg/xds/debug.go:462 +0x331 fp=0xc003517b30 sp=0xc003517978 pc=0x24024c1
istio.io/istio/pilot/pkg/xds.(*DiscoveryServer).adsz-fm(0x324a740, 0xc002d45500, 0xc009d5bc00)
        /home/howardjohn/go/src/istio.io/istio/pilot/pkg/xds/debug.go:446 +0x48 fp=0xc003517b60 sp=0xc003517b30 pc=0x241feb8
net/http.HandlerFunc.ServeHTTP(0xc00016d140, 0x324a740, 0xc002d45500, 0xc009d5bc00)
        /usr/local/go/src/net/http/server.go:2012 +0x44 fp=0xc003517b88 sp=0xc003517b60 pc=0x897764
net/http.(*ServeMux).ServeHTTP(0xc0019c42c0, 0x324a740, 0xc002d45500, 0xc009d5bc00)
        /usr/local/go/src/net/http/server.go:2387 +0x1a5 fp=0xc003517be8 sp=0xc003517b88 pc=0x8995a5
net/http.serverHandler.ServeHTTP(0xc0004f2000, 0x324a740, 0xc002d45500, 0xc009d5bc00)
        /usr/local/go/src/net/http/server.go:2807 +0xa3 fp=0xc003517c18 sp=0xc003517be8 pc=0x89aaf3
net/http.(*conn).serve(0xc019cec460, 0x3253480, 0xc003537bc0)
        /usr/local/go/src/net/http/server.go:1895 +0x86c fp=0xc003517fc8 sp=0xc003517c18 pc=0x89656c
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc003517fd0 sp=0xc003517fc8 pc=0x469521
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2933 +0x35c

Problem is here we iterate outside of the lock:

istio/pilot/pkg/xds/debug.go

Line 458 in 6ff6fce

clients := s.adsClients

howardjohn added a commit to howardjohn/istio that referenced this issue Nov 17, 2020
@howardjohn
Consistently lock and copy ads clients
08340b3 
fixes istio#28958
@howardjohn howardjohn mentioned this issue Nov 17, 2020
Merged
istio-testing pushed a commit that referenced this issue Nov 18, 2020
@howardjohn
Consistently lock and copy ads clients (#28968)
93c05a6 
* Consistently lock and copy ads clients

fixes #28958

* fix wrong reference

* optimize

* fix race
stevenctl pushed a commit to stevenctl/istio that referenced this issue Dec 17, 2020
@howardjohn
Consistently lock and copy ads clients (istio#28968)
76c0c90 
* Consistently lock and copy ads clients

fixes istio#28958

* fix wrong reference

* optimize

* fix race

(cherry picked from commit 93c05a6)
@stevenctl stevenctl mentioned this issue Dec 17, 2020
Closed
stevenctl pushed a commit to stevenctl/istio that referenced this issue Jan 5, 2021
@howardjohn
Consistently lock and copy ads clients (istio#28968)
b422b6a 
* Consistently lock and copy ads clients

fixes istio#28958

* fix wrong reference

* optimize

* fix race
istio-testing pushed a commit that referenced this issue Jan 6, 2021
@stevenctl @howardjohn @hzxuzhonghu
backport various WorkloadEntry auto-registration fixes (#29876)
1f976db 
* Consistently lock and copy ads clients (#28968)

* Consistently lock and copy ads clients

fixes #28958

* fix wrong reference

* optimize

* fix race

* Simplify internal generator and refactor a wle controller (#29554)

* Refactor internal generator and workloadentry controller

* Simplify internal gen

* fix ci

* fix lint

* fix comment

* Handle wle register/unregister race (#29604)

* handle register/unregister wle race

* update

* update tmpl

* Fix ut

* fix lint

* Fix workloadentry lost when fast reconnect (#29685)

* fix workloadentry lost when fast reconnect happen

* Handle fast reconnect： disconnect event later than reconnect event

* Cleanup when both workload/pilot down at the same time before pilot has a chance to set disconnTime

* fix ut

* address comment

* Fix leaks in workload entry controller (#29793)

* vm auto registration: discard stale disconnect events (#29691)

* implement merge patch in crd controller

* make wle auto registration robust to register unregister race

* release note

Co-authored-by: John Howard <[email protected]>
Co-authored-by: Zhonghu Xu <[email protected]>
@howardjohn howardjohn moved this to Done in Prioritization Jul 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/networking
Projects
Prioritization
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Consistently lock and copy ads clients howardjohn/istio
1 participant
@howardjohn