Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Event Sourcing might halt if CommitLogStore is not available for a long period #136

Closed
xirc opened this issue Mar 14, 2022 · 2 comments · Fixed by #137
Closed

Event Sourcing might halt if CommitLogStore is not available for a long period #136

xirc opened this issue Mar 14, 2022 · 2 comments · Fixed by #137
Labels
risk Bad things that haven't occurred yet

Comments

@xirc
Copy link
Contributor

xirc commented Mar 14, 2022

RaftActor doesn't track the progress of the event-sourcing feature. If CommitLogStore is not available for a long period (due to like journal failures), the Event Sourcing feature will halt, and it won't be recovered automatically. This halt seldom happens since CommitLogStore retries to save the committed event. Users can recover this halt by restarting each node (rolling update is possible) unless compaction deletes such events.

If compaction deletes committed entries (but they are not persisted to CommitLogStore), it is difficult to recover since RaftActor doesn't know which committed entry to persist to CommitLogStore.

Related code

@xirc xirc added the risk Bad things that haven't occurred yet label Mar 14, 2022
@xirc
Copy link
Contributor Author

xirc commented Mar 14, 2022

Lerna team will address this issue by the following:

  • RaftActor tracks the progress of Event Sourcing (CommitLogStore).
  • Compaction doesn't delete entries that are not persisted to CommitLogStore yet.

@xirc
Copy link
Contributor Author

xirc commented Mar 18, 2022

#142 addresses some additional challenges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
risk Bad things that haven't occurred yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant