Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(store): use symlinks for empty blocks heighs #3747

Merged
merged 1 commit into from
Sep 18, 2024

Conversation

walldiss
Copy link
Member

There is limitation on max amount of hard links in ext4 filesystem of 65000. Some chains have more empty blocks (arabica). We will use symlinks for empty blocks to avoid this limitation, while keeping hard-links for non-empty blocks.

@walldiss walldiss added kind:fix Attached to bug-fixing PRs shwap labels Sep 18, 2024
@walldiss walldiss self-assigned this Sep 18, 2024
@walldiss walldiss changed the base branch from main to shwap September 18, 2024 18:39
@walldiss walldiss force-pushed the use-symlinks-for-emptyheigh branch from 98b5ab2 to c711d65 Compare September 18, 2024 18:40
@codecov-commenter
Copy link

codecov-commenter commented Sep 18, 2024

Codecov Report

Attention: Patch coverage is 77.77778% with 2 lines in your changes missing coverage. Please review.

Please upload report for BASE (shwap@89e8781). Learn more about missing BASE report.

Files with missing lines Patch % Lines
store/store.go 77.77% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             shwap    #3747   +/-   ##
========================================
  Coverage         ?   46.91%           
========================================
  Files            ?      314           
  Lines            ?    17867           
  Branches         ?        0           
========================================
  Hits             ?     8383           
  Misses           ?     8476           
  Partials         ?     1008           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

store/store.go Outdated
@@ -170,11 +170,18 @@ func (s *Store) createFile(
return false, nil
}

func (s *Store) linkHeight(datahash share.DataHash, height uint64) error {
Copy link
Member

@Wondertan Wondertan Sep 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about keeping this method usage as is, but change it to check for empty root inside and make needed link accordingly with proper comment. In other words, decision on whether its symlink or hardlink stays an impl detail of linkHeight method

@walldiss walldiss force-pushed the use-symlinks-for-emptyheigh branch from c711d65 to de00998 Compare September 18, 2024 20:18
@walldiss walldiss merged commit 2c67306 into celestiaorg:shwap Sep 18, 2024
30 checks passed
@walldiss walldiss deleted the use-symlinks-for-emptyheigh branch September 18, 2024 20:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:fix Attached to bug-fixing PRs shwap
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants