New Tutorial: Storing and Retrieving Data on Filecoin via IPFS

[terichadbourne]

In discussing potential Filecoin content with @mishmosh, she suggested a tutorial on how Filecoin enhances IPFS, the target audience of which would be existing IPFS users who are new to Filecoin.

Some notes to include:

• they share CIDs
• you can create a storage deal that points to an IPFS CID as the data source, rather than having to supply your data directly to the storage miner

My understanding of the interoperability is quite limited, so I'd love further suggestions on ways an IPFS user might step things up a notch with Filecoin.

[lidel]

There are more specific variants of this question, but generic one that I hear over and over is:

• How can I leverage the existence of Filecoin to keep specific CID alive and available on IPFS for a specific amount of time?

[terichadbourne]

@lidel Great question! Does "alive and available" to you mean that anyone can access it if they have the CID, or just that you can access it if you have the CID? I'm assuming the way the retrieval market works might throw a wrench in the former.

[lidel]

@terichadbourne The nuance you described is exactly why I mentioned this question: users are not aware of the complexity involved and assume "Filecoin node itself will just make CIDs available to everyone on the IPFS network".

In practice, to "make CID available to everyone on IPFS" it is enough for me to be able to "fetch data behind a CID from Filecoin back to my IPFS node" – at that point my IPFS node starts providing retrieved data to the rest of the IPFS swarm.

I noticed this indirection is surprising to IPFS users: perhaps this additional step should be clearly stated somewhere, eg. as an example following the description of retrieval market?

[terichadbourne]

For reference, note the new IPFS & Filecoin section in the Filecoin docs: https://docs.filecoin.io/introduction/ipfs-and-filecoin/

[terichadbourne]

@pooja may tentatively have time to take the lead on this late in Q3 after launch (🙏), as she and @momack2 will both be gaining a lot of insight in the coming weeks into user needs and messaging of interoperability. In the meantime, let's continue to use this issue as a place to note questions we'd like to answer for users or documents created elsewhere that are helpful for explaining the relationship between IPFS and Filecoin.

[terichadbourne]

@mishmosh has offered to write up a proposed outline for this one to kick off more discussion. ❤️

I also just noticed some tidbits on the IPFS/Filecoin connection that may prove useful in ipfs/notes#435

[terichadbourne]

After more discussion it does seem like the the follow-up piece discussed in the DAG/CAR Files proposal (#412) could be this one. Notes from a meeting between Mosh, Celeste and I can be found here, and the notes for the DAG/CAR tutorial provide additional context.

@mitchwagner will be refining an outline for this one while he drafts the DAG/CAR File tutorial.

[terichadbourne]

@ribasushi We've had some previous conversations about a potential tutorial to follow our newly released Merkle DAGs tutorial, the subject of which would be more or less how to get files in and out of Filecoin via IPFS. Intended target would be primarily IPFS users curious about interoperability. This would build on the knowledge gained in the Merkle DAG tutorial. If I recall correctly, you may have said that Filecoin users will no longer actually be exposed to the fact that they're using CAR files, as we had previously envisioned. Are you in a position to tell us how clear the storage/retrieval process is now and how close we are to being able to tell the story? If feasible, it would be awesome to have a stab at an extremely rough outline of the concepts we'd want to cover. (There's an old one from Celeste in the 2nd part of this comment but I'm not sure how out of date it is. Note also some other notes docs linked from the previous comment here.)

There's some debate in Slack about the desire to have a blog post or other written piece describing the differences between IPFS and Filecoin, and I don't understand the issue enough to know whether the same objections would apply to our plan to tell the story of how you can store and retrieve data on Filecoin via IPFS. Perhaps we can draft a rough outline to demonstrate what we have in mind and then stop and get a green light before proceeding.

[mikeal]

I think this tutorial is a little bit too broad to make headway on in the short term. If you’re just talking about storing and retrieving files you have some clear workflows, but once you get into structured data there’s going to be big differences in these workflows depending on the use case. For instance “Storing and Retrieving Archival Data” could be a tutorial because that’s narrow enough to have much more specific recommendations and maybe even some features to help it along.

We should probably wait until after the Web3 use cases get surfaced and prioritized before deciding exactly what to go after here.

[terichadbourne]

Thanks for the input, @mikeal. Totally understand why it would be easier to pick a specific use case and describe the approach in that context.

For existing IPFS users (not all Filecoin users generally), does storing/retrieving files on Filecoin via IPFS feel like a less common use case than storing/retrieving various types of non-file data?