feat: introduce block meta cache #17360
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
We previously had a block meta cache, but it was rather coarse-grained. Blockmetas were cached at the segment level, which worked reasonably for scenarios like general table pruning. However, for operations like
replace into
, we needed something more fine-grained.So, in this PR:
Rename the old segment-level block meta cache to
SegmentBlockMetasCache
.Each item in this cache is an array of BlockMeta objects, representing all the block metas for a specific segment.
Introduce a new
BlockMetaCache
.Each item in this cache is a single
BlockMeta
.Adjust
system.caches
andsystem$set_cache_capacity
to fit these changes.Tests
Type of change
This change is![Reviewable](https://camo.githubusercontent.com/1541c4039185914e83657d3683ec25920c672c6c5c7ab4240ee7bff601adec0b/68747470733a2f2f72657669657761626c652e696f2f7265766965775f627574746f6e2e737667)