Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a synchronous "before checkpoint" hook? #12

Open
bjchambers opened this issue Nov 6, 2023 · 1 comment
Open

Add a synchronous "before checkpoint" hook? #12

bjchambers opened this issue Nov 6, 2023 · 1 comment

Comments

@bjchambers
Copy link

For some use cases of a write ahead log, I'd like to keep an in-memory representation (which gets recovered) of the contents of the log. When a checkpoint is made, I can drop the check-pointed information from memory since it is now part of the checkpoint.

One difficulty, as far as I can tell, is that the checkpoint method is called in a separate thread -- this means the following could happen:

  1. Commit some data A to the log
  2. Initiate a checkpoint
  3. Commit some data B to the log
  4. Actually perform the checkpoint

The problem I have is that at that point, the contents of A and B are in the in-memory representaiton, while I only want to checkpoint (and potentially drop) A. For my use case, I'm fine keeping the in-memory representation behind a Mutex (there should be only one writer to the log), which also simplifies a lot of the logic.

There are a few things that could help me:

  1. If the result of committing data indicated that a checkpoint would be taken. I think this would only help if there was a lock, but then I could detect that a checkpoint would be taken after A, and arrange to have a separate in-memory representation started for data added after that.
  2. If there was a "pre-checkpoint" hook that was called before other commits were allowed. Again, this would allow me to split the in-memory representation for B.
@ecton
Copy link
Member

ecton commented Nov 7, 2023

Thank you for the idea for the additional use case for this crate. I think the easiest thing to offer would be your second suggestion.

If you need this quickly, I would love a pull request. Otherwise I will eventually be getting back onto this project, but right now it's not my primary focus.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants