Non-leaders passivate entities eventually #200
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 4 commits
April 7, 2023 18:07
A non-leader (follower or candidate) asks permission to passivate an entity to the leader. The leader replies with a permit or denial based on whether the leader's entity is running. If the non-leader receives a permit, it will passivate its entity. This mechanism ensures that non-leaders can passivate their entity eventually after the leader passivate the leader's entity.
The leader could permit passivation mistakenly if the entity has been terminated with a failure. The leader will recover from the mistake by requesting non-leaders such that non-leaders entities start.
negokaz
reviewed
Apr 12, 2023
core/src/main/scala/lerna/akka/entityreplication/raft/RaftActor.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/lerna/akka/entityreplication/raft/Leader.scala
Outdated
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #199
Solution
A non-leader (follower or candidate) asks permission to passivate an entity to the leader. The leader replies with a permit or denial based on whether the leader's entity is running. If the non-leader receives a permit, it will passivate its entity. This mechanism ensures that non-leaders can passivate their entity eventually after the leader passivate the leader's entity.
There is a subtle timing at which the leader could permit passivation mistakenly (for example, the leader's entity is terminated with an unexpected failure). To address this leader's mistake, when the leader restarts its entity, the leader makes a request to non-leaders such that non-leaders entities start.
Considerations
Since old nodes cannot deserialize newly introduced remote messages, this mechanism doesn't work during a rolling update. If an entity uses ReceiveTimeout to passivate itself (please refer: https://github.com/lerna-stack/akka-entity-replication/blob/v2.2.0/docs/typed/implementation_guide.md#passivation), it will be passivated eventually after the rolling update completes.