feature: implement hamt writing #603
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Closed
rklaehn
force-pushed
the
hamt-writing-2
branch
from
70b39d0 to
260c0c0
Compare
rklaehn
force-pushed
the
hamt-writing-2
branch
from
260c0c0 to
b0f3007
Compare
The real fix is of course to make this a streaming request... But that will come later.
rklaehn
force-pushed
the
hamt-writing-2
branch
from
124143a to
64331d6
Compare
# Conflicts: # iroh-unixfs/src/builder.rs
rklaehn
commented
Dec 16, 2022
| stream! { | ||
| while let Some(block) = blocks.next().await { | ||
| let block = block?; | ||
| let block_size = block.data().len() as u64; | ||
| let block_size = block.data().len() as u64 + block.links().len() as u64 * 128; |
There was a problem hiding this comment.
@dignifiedquire this is the temp fix for not having blocks that are too large, until we implement https://github.com/n0-computer/iroh/issues/594
|
Update: go-ipfs interop does work, at least for my big test dir:
|
I could not find an actual collision, so I just added twice the exact same name for a test, which currently has the same effect.
| pub fn encode<'a>(self) -> LocalBoxStream<'a, Result<Block>> { | ||
| match self { | ||
| Self::Branch(tree) => { | ||
| async_stream::try_stream! { |
There was a problem hiding this comment.
This reads quite sane, great job.
dignifiedquire
approved these changes
Dec 16, 2022
ppodolsky
pushed a commit
to izihawa/iroh
that referenced
this pull request
Dec 28, 2022
* feature: implement hamt writing * fix: consider link size when sending data The real fix is of course to make this a streaming request... But that will come later. * test: add test for a truly large directory * refactor: cleanup * refactor: properly handle hash collisions I could not find an actual collision, so I just added twice the exact same name for a test, which currently has the same effect.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This provides basic hamt writing. You can add a big directory, get it back, and look at it from the gateway.
Go-ipfs interop has been tried for one large dir and works.
I have manually added both large non hamt dirs and large hamt dirs and looked at them in the gw.