Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(opensearch): Rollover usage events at a file size rather than time-based manner #10006

Merged
merged 1 commit into from
Mar 7, 2024

Conversation

darnaut
Copy link
Collaborator

@darnaut darnaut commented Mar 7, 2024

The rollover operation rolls an alias over to a new index when the managed index meets one of the rollover conditions. Since the rollover conditions are min_index_age: 1d and min_size: 5gb conditions, the index ends up being rolled over every day since the minimum index age condition will evaluate to true.

Instead of time-based rollover, this changes the rollover condition to only be based on the minimum size.

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions github-actions bot added devops PR or Issue related to DataHub backend & deployment community-contribution PR or Issue raised by member(s) of DataHub Community labels Mar 7, 2024
Copy link
Collaborator

@david-leifker david-leifker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs schema_version increment

@darnaut darnaut force-pushed the darnaut-opensearch-usage-event-rollover branch from a858326 to df17ce3 Compare March 7, 2024 16:40
@darnaut darnaut merged commit c4a4532 into master Mar 7, 2024
36 checks passed
@darnaut darnaut deleted the darnaut-opensearch-usage-event-rollover branch March 7, 2024 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community-contribution PR or Issue raised by member(s) of DataHub Community devops PR or Issue related to DataHub backend & deployment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants