Tracks progress of event sourcing #137
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 9 commits
March 14, 2022 14:21
AppendCommittedEntries and its reply message AppendCommittedEntriesResponse allows * RaftActor can request saving multiple committed log entries to CommitLogStoreActor at one message. * RaftActor can fetch the last index of committed log entries that CommitLogStoreActor saved.
This feature enables the compaction won't delete committed entries that are not persisted to the event sourcing store (aka CommitLogStore) yet.
… entries Even if the compaction continues, the remaining entries will trigger new compaction at the next tick. This repeated compaction might consume unnecessary resources.
This sending is not needed since RaftActor sends CommitLogStoreActor.AppendCommittedEntries periodically instead. RaftActor will retry failed AppendCommittedEntries, which makes RaftActorStore.save, and the related settings and classes unnecessary.
* Remove inlined settings for maintainability * Remove raft.eventsourced.commit-log-store settings
It's impossible to contain too-many entries into a single AppendCommittedEntries since one message size is limited. To address this limitation, RaftActor splits entries into batches and sends a AppendCommittedEntries per each batch.
CommitLogStore and ShardedCommitLogStore can be removed since they are package private.
Use CommitLogStoreActor.AppendCommittedEntries instead.
xirc
force-pushed
the
event-sourcing-progress-tacking
branch
from
e9a1d42 to
01754e0
Compare
added 6 commits
March 15, 2022 09:54
…-threshold If preserve-log-size equals to or is greater than log-size-threshold, compaction won't happen.
…alArgumentException explicitly
Add event-sourcing progress tracking feature.
log-size-threshold should be larger than preserve-log-size.
negokaz
reviewed
Mar 16, 2022
src/main/scala/lerna/akka/entityreplication/raft/eventsourced/CommitLogStoreActor.scala
Outdated
Show resolved
Hide resolved
src/main/scala/lerna/akka/entityreplication/raft/eventsourced/CommitLogStoreActor.scala
Outdated
Show resolved
Hide resolved
xirc
commented
Mar 16, 2022
src/test/scala/lerna/akka/entityreplication/raft/RaftActorLeaderEventSourcingSpec.scala
Outdated
Show resolved
Hide resolved
negokaz
reviewed
Mar 16, 2022
added 2 commits
March 16, 2022 14:30
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #136
Overview
RaftActorwill track a log index (namedeventSourcingIndex). This index indicates thatCommitLogStoreActor(which is responsible for Event Sourcing) already persisted events with indices lower than or equal to that index. This index enablesRaftActorto know the progress of Event Sourcing. This index is periodically exchanged by the new messages (CommitLogStore.AppendCommittedEntriesandCommitLogStore.AppendCommittedEntriesResponse) betweenRaftActorandCommitLogStoreActor.RaftActor(only the leader) checks new committed entries (which index should >eventSourcingIndexand <=commitIndex) at a certain period (namedcommitted-log-entries-check-intervalorEventSourcingTick) and will send those entries (included toCommitLogStore.AppendCommittedEntries) if available. This periodically checking mechanism prevents halting Event Sourcing. It also allowsRaftActorto retry committed entries persisting (the previous persisting failed for some reason like journal failures).Using
eventSourcingIndexRaftActorcan know what entry can be removed safely on compaction. More precisely, committed entries that are persisted to the event-sourcing store can be removed safely. By this PR, compaction won't delete entries that are not persisted to the event-sourcing store yet.By introducing event soucing progress tracking, followers and candidates don't have to send committed entries to the event-sourcing store. This PR removes such sending, which might reduce the required network resource.