Streamed pins #290
Streamed pins #290
Conversation
|
Thank you for submitting this PR!
Getting other community members to do a review would be great help too on complex PRs (you can ask in the chats/forums). If you are unsure about something, just leave us a comment.
We currently aim to provide initial feedback/triaging within two business days. Please keep an eye on any labelling actions, as these will indicate priorities and status of your contribution. |
See ipfs/boxo#290 This PR follow the changes in the Pinner to make listing recursive and direct pins asynchronous, which in turns allow pin/ls to build and emit results without having to wait anything, or accumulate too much in memory. Note: there is a tradeoff for pin/ls?type=all: - keep the recursive pins in memory (which I chose) - ask the pinner twice for the recursive pins, and limit memory usage Also, follow the changes in the GC with similar benefit of not having to wait the full pin list. Add a test. Also, follow the changes in pin.Verify.
Codecov Report
@@ Coverage Diff @@
## main #290 +/- ##
==========================================
+ Coverage 49.51% 49.56% +0.04%
==========================================
Files 282 283 +1
Lines 33836 33876 +40
==========================================
+ Hits 16755 16791 +36
- Misses 15316 15322 +6
+ Partials 1765 1763 -2
|
See ipfs/boxo#290 This PR follow the changes in the Pinner to make listing recursive and direct pins asynchronous, which in turns allow pin/ls to build and emit results without having to wait anything, or accumulate too much in memory. Note: there is a tradeoff for pin/ls?type=all: - keep the recursive pins in memory (which I chose) - ask the pinner twice for the recursive pins, and limit memory usage Also, follow the changes in the GC with similar benefit of not having to wait the full pin list. Add a test. Also, follow the changes in pin.Verify.
|
@MichaelMure : I know this has been open for awhile. Apologies. Maintainers spoke on 2023-05-25. We'll make sure this goes out with Kubo 0.21. Slack context: https://filecoinproject.slack.com/archives/C03L0G3B4RX/p1684507196595209 |
MichaelMure
force-pushed
the
streamed-pins
branch
from
9207501 to
edd7540
Compare
See ipfs/boxo#290 This PR follow the changes in the Pinner to make listing recursive and direct pins asynchronous, which in turns allow pin/ls to build and emit results without having to wait anything, or accumulate too much in memory. Note: there is a tradeoff for pin/ls?type=all: - keep the recursive pins in memory (which I chose) - ask the pinner twice for the recursive pins, and limit memory usage Also, follow the changes in the GC with similar benefit of not having to wait the full pin list. Add a test. Also, follow the changes in pin.Verify.
|
@Jorropo what's the status? |
The rational is that if the pin list get big, a synchronous call to get the complete list can delay handling unnecessarily. For example, when listing indirect pins, you can start walking the DAGs immediately with the first recursive pin instead of waiting for the full list. This matters even more on low power device, of if the pin list is stored remotely. Replace: - ipfs/go-ipfs-pinner#24 - ipfs/go-ipfs-provider#48
Rationale is that generating a test DAG is quite difficult, and anything that helps writing better tests is helpful. Replace ipfs/go-ipfs-blocksutil#18
See ipfs/boxo#290 This PR follow the changes in the Pinner to make listing recursive and direct pins asynchronous, which in turns allow pin/ls to build and emit results without having to wait anything, or accumulate too much in memory. Note: there is a tradeoff for pin/ls?type=all: - keep the recursive pins in memory (which I chose) - ask the pinner twice for the recursive pins, and limit memory usage Also, follow the changes in the GC with similar benefit of not having to wait the full pin list. Add a test. Also, follow the changes in pin.Verify.
See ipfs/boxo#290 This PR follow the changes in the Pinner to make listing recursive and direct pins asynchronous, which in turns allow pin/ls to build and emit results without having to wait anything, or accumulate too much in memory. Note: there is a tradeoff for pin/ls?type=all: - keep the recursive pins in memory (which I chose) - ask the pinner twice for the recursive pins, and limit memory usage Also, follow the changes in the GC with similar benefit of not having to wait the full pin list. Add a test. Also, follow the changes in pin.Verify.
Corresponding kubo PR: ipfs/kubo#9859
The rational is that if the pin list get big, a synchronous call to get the complete list can delay handling unnecessarily. For example, when listing indirect pins, you can start walking the DAGs immediately with the first recursive pin instead of waiting for the full list.
This matters even more on low power device, of if the pin list is stored remotely.
Also in this PR, a DAG generator to simplify testing (used in the kubo PR to test the GC).
replace ipfs/interface-go-ipfs-core#109
replace ipfs/go-ipfs-pinner#24
replace ipfs/go-ipfs-provider#48
replace ipfs/go-ipfs-blocksutil#18