Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(etl): add stream length checks to prevent overflow #92

Merged
merged 1 commit into from
Feb 10, 2025

Conversation

armyhaylenko
Copy link
Collaborator

This pr:

  1. adds a stream counter to the snapshot binary, which counts how many consecutive messages were sent
  2. when the stream counter reaches some constant value (20M in our case), check accounts stream length to be below the desired value
  3. if it's above the desired value, sleep in 1-second intervals until the stream length drops below the desired value
  4. reset the stream counter and send another batch of accounts to the redis instance.

n00m4d
n00m4d previously approved these changes Feb 10, 2025
@armyhaylenko armyhaylenko merged commit ce6a79d into main Feb 10, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants