Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend OH SQL for snapshots retention #223

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

teamurko
Copy link
Collaborator

@teamurko teamurko commented Oct 8, 2024

Summary

Support configurable snapshots retention (SQL, PR1)

  • time-based retention, supported hour and day granularity
  • count-based retention, the most recent N are preserved when expiring snapshots

Next:

  • Handle snapshotsRetention property
  • Propagate the policy to ES app

Changes

  • Client-facing API Changes
  • Internal API Changes
  • Bug Fixes
  • New Features
  • Performance Improvements
  • Code Style
  • Refactoring
  • Documentation
  • Tests

For all the boxes checked, please include additional details of the changes made in this pull request.

Testing Done

  • Manually Tested on local docker setup. Please include commands ran, and their output.
  • Added new tests for the changes made.
  • Updated existing tests to reflect the changes made.
  • No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
  • Some other form of testing like staging or soak time in production. Please explain.

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.

Additional Information

  • Breaking Changes
  • Deprecations
  • Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

mockTableService.enqueue(mockResponse(200, existingOpenhouseTable)); // doRefresh()
mockTableService.enqueue(mockResponse(201, tableAfterSetSnapshotsRetention)); // doCommit()
String ddlWithSchema =
"ALTER TABLE openhouse.dbSetSnapshotsRetention.t1 SET POLICY (SNAPSHOTS_RETENTION TTL=24H COUNT=10)";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • shall we use something more simple like version instead of snapshots, the latter was a overloaded term that can also be used to describe a non-partitioned table.
  • alternatively what about SET POLICY RETAIN 100 HOURS SET POLICY RETAIN 50 VERSIONS

@autumnust
Copy link
Collaborator

I know delta lake has something called VACCUM being similar. Maybe we can get some inspiration from Polaris for this syntax.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants