[fix][metadata] Don't execute Bookkeeper metadata callbacks on Zookeeper event thread #17620
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
eolivelli
reviewed
Sep 13, 2022
...r-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/PulsarRegistrationClient.java
Show resolved
Hide resolved
merlimat
approved these changes
Sep 14, 2022
|
@lhotari Do you have the stack trace for the deadlock to attach here for context? |
coderzc
approved these changes
Sep 14, 2022
hezhangjian
approved these changes
Sep 14, 2022
@merlimat I only had a heap dump of the problem so I'm not sure about the deadlock details. The heap dump includes a thread dump, but without lock info. Here's the full thread dump: Here's a subset of threads related to metadata store and ZK. There might be several issues that are causing the issue. Here are the metadata & ZK related threads from the full thread dump: |
4 tasks
|
I created a separate PR about handling session events in a separate thread: #17638 |
eolivelli
pushed a commit
to datastax/pulsar
that referenced
this pull request
Sep 14, 2022
…per event thread (apache#17620) (cherry picked from commit 8e0ae80)
4 tasks
Technoboy-
pushed a commit
that referenced
this pull request
Sep 26, 2022
…per event thread (#17620)
nicoloboschi
pushed a commit
to datastax/pulsar
that referenced
this pull request
Sep 28, 2022
…per event thread (apache#17620) (cherry picked from commit 8e0ae80) (cherry picked from commit f764350)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
In a Pulsar 2.10.x test environment we ran into an issue where there were 12.3 million ZK WatchEvents in the heap dump.
The ZK Event thread was blocked and the ZK session had also been expired.
Modifications
Run callbacks on a separate thread.
Additional context
PIP-45 changes #12770 and #13296
Documentation
doc-required(Your PR needs to update docs and you will update later)
doc-not-needed(Please explain why)
doc(Your PR contains doc changes)
doc-complete(Docs have been already added)