-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
opt(rollup): change the way rollups are done #7253
Conversation
…ibrahim/delta-log
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. @NamanJain8 is the disk size bounded after this change?
This PR needs a proper title and description. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 1 of 3 files at r5, 1 of 1 files at r6.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @ahsanbarkati, @jarifibrahim, @martinmr, and @vvbalaji-dgraph)
posting/mvcc.go, line 136 at r6 (raw file):
doRollup := func(batch *[][]byte, priority int) { currTs := time.Now().Unix()
forcerollup is good to have. Use it over the priority pool.
Maybe add a TODO about dealing with opRollup not running.
Yes, it is. Mentioned that in PR description. |
ISSUE: In a load that has a lot of updations to a key, a lot of deltas accumulate over time. These keys are added to the batch for a rollup. Due to the lossy behavior of rollups, they get dropped while the dgraph is busy rolling up keys (that might have much smaller delta count). FIX: We now add keys to rollupBatch just after pushing the deltas over to badger. We now have a priority inside the keys that we rollup. The keys for which the deltas have increased up to a certain limit (hard-coded 500), we add it to high priority rollupKeyPool so that it has high chances to get rolled up and does not get dropped due to lossy behavior of rollups. Co-authored-by: Manish R Jain <[email protected]> Co-authored-by: NamanJain8 <[email protected]> (cherry picked from commit cbdc991)
ISSUE: In a load that has a lot of updations to a key, a lot of deltas accumulate over time. These keys are added to the batch for a rollup. Due to the lossy behavior of rollups, they get dropped while the dgraph is busy rolling up keys (that might have much smaller delta count). FIX: We now add keys to rollupBatch just after pushing the deltas over to badger. We now have a priority inside the keys that we rollup. The keys for which the deltas have increased up to a certain limit (hard-coded 500), we add it to high priority rollupKeyPool so that it has high chances to get rolled up and does not get dropped due to lossy behavior of rollups. Co-authored-by: Manish R Jain <[email protected]> Co-authored-by: NamanJain8 <[email protected]> (cherry picked from commit cbdc991) Co-authored-by: Ibrahim Jarif <[email protected]>
Fixes DGRAPH-2608
ISSUE:
In a load that has a lot of updations to a key, a lot of deltas accumulate over time. These keys are added to the batch for a rollup. Due to the lossy behavior of rollups, they get dropped while the dgraph is busy rolling up keys (that might have much smaller delta count).
Fix:
This PR changes the way rollups are done.
rollupKeyPool
so that it has high chances to get rolled up and does not get dropped due to lossy behavior of rollups.Results:
With a workload where a set of values are frequently updated:
deltaCount
increases to 7-9k for lots of keys.This change is