This repository has been archived by the owner on Aug 12, 2020. It is now read-only.
exporter: Support slicing streams stored in deeply nested DAGs #208
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ghost
assigned
achingbrain
commented
Mar 27, 2018
| @@ -75,6 +74,7 @@ | |||
| "pull-pause": "0.0.2", | |||
| "pull-pushable": "^2.2.0", | |||
| "pull-stream": "^3.6.2", | |||
| "pull-through": "^1.0.18", | |||
There was a problem hiding this comment.
This PR doesn't actually use pull-through - it appears to be a missing dependency as it's used by the builder and trickle-reducer modules.
daviddias
suggested changes
Mar 27, 2018
test/exporter.js
Outdated
| console.info(err) | ||
|
|
||
| console.info(data.length) | ||
| console.info(expected.length) |
There was a problem hiding this comment.
Extra console logs for testing?
There was a problem hiding this comment.
I've removed them
achingbrain
force-pushed
the
slice-nested-streams
branch
from
bf62319 to
6e6e39f
Compare
daviddias
approved these changes
Mar 27, 2018
achingbrain
added a commit
that referenced
this pull request
May 26, 2018
Part of making the hases emitted by js-ipfs the same as go-ipfs meant unsetting the data attribute of UnixFS metadata objects if the data buffer had 0 length. ipfs.files.get converts a pull stream of files into a readable stream of files and all of the content pull streams too. If an empty buffer is not emitted by the file content pull stream for file with no data then no file is written out to the filesystem. This was a fun bug to track down. This module used to emit empty buffers for UnixFS metadata with no data property but I removed it as part of #208 as there was no test that failed without it so here we reinstate it and add a test.
achingbrain
added a commit
that referenced
this pull request
Jun 8, 2018
Part of making the hases emitted by js-ipfs the same as go-ipfs meant unsetting the data attribute of UnixFS metadata objects if the data buffer had 0 length. ipfs.files.get converts a pull stream of files into a readable stream of files and all of the content pull streams too. If an empty buffer is not emitted by the file content pull stream for file with no data then no file is written out to the filesystem. This was a fun bug to track down. This module used to emit empty buffers for UnixFS metadata with no data property but I removed it as part of #208 as there was no test that failed without it so here we reinstate it and add a test.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The previous attempt at this didn't handle larger files where the tree can have multiple tiers of DAGNodes - this PR builds on the existing exporter tree traversal to export slices of streams instead of the whole file (the slice could be the same size of the file of course).
The approach is:
beginandendare passed, sanitise & respect them, otherwise use0and the filesize reported by the root DAGNode to export the whole fileblockSizesattribute of the DAGNode to work out which DAGLinks we should follow to only retrieve the requested byte rangeblockSizesattribute to only traverse to the requested data.