Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tool: support sst dump in risectl #3101

Closed
hzxa21 opened this issue Jun 9, 2022 · 4 comments
Closed

tool: support sst dump in risectl #3101

hzxa21 opened this issue Jun 9, 2022 · 4 comments
Labels
component/storage Storage difficulty/medium Issues that need some knowledge of the whole system good first issue Good for newcomers help wanted Issues that need help from contributors

Comments

@hzxa21
Copy link
Collaborator

hzxa21 commented Jun 9, 2022

#2640 #782 add support for list-kv pairs in risectl. It is also beneficial for debugging to support sst dump.

Basic

We can start by adding support to show the following information via sst dump:

  • SstableIdInfo stored in HummockVersion (persisted in meta store)
    • Create timestamp
    • Delete timestamp
  • SstableInfo stored in HummockVersion (persisted in meta store)
    • Key range
    • VNodes
    • Prefixes
  • SstableMeta persisted in object store.
    • Checksum
    • Block meta (compression algo, size, ...)
    • Bloom filter size
    • Key count
    • Version
    • ...

Advanced

We can support the following advanced usage when the basics are done:

  • Verify checksum of a SST
  • Iterate kv paris in a SST
    - Print kv paris in a human readble format (show table id, epoch, user key, user value)
    - Interpret kv paris based on the table schema
  • Dump raw data of SST into a file
  • Regenerate and comapre SST with different configurations (block size, compression, filter policy, ...)
@hzxa21 hzxa21 added good first issue Good for newcomers help wanted Issues that need help from contributors component/storage Storage labels Jun 9, 2022
@Graphcalibur Graphcalibur self-assigned this Jun 14, 2022
@Graphcalibur Graphcalibur removed their assignment Jun 21, 2022
@Graphcalibur
Copy link
Contributor

All the information under "Basic" is now displayed when the sst-dump command is called.

@skyzh
Copy link
Contributor

skyzh commented Jun 21, 2022

That's cool! @hzxa21 shall we also implement the advanced part? I believe it is not necessary at current stage.

@hzxa21
Copy link
Collaborator Author

hzxa21 commented Jun 21, 2022

Cool. We can revisit the advanced parts when needed in the future.

@ALeitert
Copy link
Contributor

  • Verify checksum of a SST

This is already done in SstableMeta::decode() and Block::decode().

@ALeitert ALeitert removed their assignment Jul 17, 2022
@skyzh skyzh added the event/OSD label Sep 2, 2022
@skyzh skyzh added difficulty/medium Issues that need some knowledge of the whole system and removed event/OSD labels Sep 16, 2022
@hzxa21 hzxa21 closed this as completed Jun 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/storage Storage difficulty/medium Issues that need some knowledge of the whole system good first issue Good for newcomers help wanted Issues that need help from contributors
Projects
None yet
Development

No branches or pull requests

4 participants