Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement] limit persistent index compaction by disk (backport #36681) #37693

Merged
merged 4 commits into from
Dec 25, 2023
Merged

[Enhancement] limit persistent index compaction by disk (backport #36681) #37693

merged 4 commits into from
Dec 25, 2023

Commits on Dec 25, 2023

  1. [Enhancement] limit persistent index compaction by disk (#36681) 

    Why I'm doing:
In current implementation, we doesn't control the compaction concurrency on one disk, which will cause compaction unbalanced between disks, and some disk's IO cost will be too high.

What I'm doing:

Limit the disk compaction concurrency by config pindex_major_compaction_limit_per_disk .
Refactor persistent index compaction scheduler code.

Signed-off-by: luohaha <[email protected]>
(cherry picked from commit a44147f)

# Conflicts:
#	be/src/storage/persistent_index.h
#	be/src/storage/persistent_index_compaction_manager.cpp
    @luohaha @mergify
    luohaha authored and mergify[bot] committed Dec 25, 2023
    Configuration menu
    Copy the full SHA
    cc5f4f3 View commit details
    Browse the repository at this point in the history

  2. resolve conflict 

    Signed-off-by: Yixin Luo <[email protected]>
    @luohaha
    luohaha committed Dec 25, 2023
    Configuration menu
    Copy the full SHA
    8879701 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'branch-3.2' into mergify/bp/branch-3.2/pr-36681 

    Signed-off-by: Yixin Luo <[email protected]>
    @luohaha
    luohaha authored Dec 25, 2023
    Configuration menu
    Copy the full SHA
    961be5c View commit details
    Browse the repository at this point in the history

  4. fix ut 

    Signed-off-by: Yixin Luo <[email protected]>
    @luohaha
    luohaha committed Dec 25, 2023
    Configuration menu
    Copy the full SHA
    0344581 View commit details
    Browse the repository at this point in the history