Open Discussion on Programmatic Storage with FVM

[trruckerfling]

Simple Summary:

At the recent Fil Dev Summit Singapore (FDS), the FVM team ran a FVM app & tooling track to open up the discussion around FVM's existing features, current & future roadmap and user feedback/ideas. The track covered 3 main categories:

• Programmatic Storage
• DeFi & dApp development
• Protocol-level design

Each category had a dedicated discussion to it where FVM team, Filecoin core devs, partners and community builders participated. This article is part of a series of discussions that highlight takeaways from the event, to continue and open up the conversation online. You can easily navigate to other discussions in the hyperlinks above.

Motivation and ask from community:

FVM builders and partners, including Filecoin devs are invited to participate in these discussions, below in thread ⬇️ and give direct feedback to FVM product and engineering teams, to build a valuable VM. These discussions are meant to be ongoing, from FDS Singapore to Iceland and beyond.

Programmatic storage on Filecoin with FVM discussion:

1. With FVM enabling programmatic storage on Filecoin, who takes on the ownership for data verification, provenance and accountability?

Context: 'Aggregators' refer to platforms that take in data storage requests, of any size, from clients. Aggregators then use their built-in deal making engines to make deals to store that data on Filecoin, while returning a confirmation of that stored data to clients. An example of an aggregator is lighthouse.storage

Key discussion points:

• Does the aggregator, client or SP take on the responsibility for malicious data? The current opinion is for aggregators to have their clients take ownership for data uploaded. If data turns out to be malicious, the aggregator reserves the right to take down the data. (contributor: @galen-mcandrew)

• There is a potential gap in the programmatic storage market (PSM) for 'authenticators' to verify and authenticate clients' data. Some ideas discussed include aggregators or storage providers (SP), only taking in clients' data that has been verified by authenticators. This was discussed to be separate from FIL+. There could be several credible authenticators in the Filecoin PSM, that can offer differentiated authenticated services for different users' needs. Authenticators will also be decentralized e.g. a network of validators to verify the data. This idea puts the ownership of data provenance and authenticity on the client and incentivizes them to provide quality data, in order to store data. (contributors: @galen-mcandrew @raulk @longfeiWan9 @nandit123 @trruckerfling )

  • Current action: FVM product team is working on a design for an authenticator and aim to provide a standard specs, to enable interested ecosystem partners to build from it and provide the service.

2. Programmatic storage markets (PSM) in Filecoin today, does not provide a clear user journey to get data stored. How do we simplify that?

Context: There are currently various ways to store data with FVM-enabled workflows, including direct deal making (client→SP) and aggregation (client→aggregator→SP). This is in addition to the ability to pin data to IPFS, which is typically used in complement with FVM-enabled workflows (e.g. data is stored on Filecoin and a copy is pinned to IPFS for easy retrievals). There are also various data size requirements to take note of, depending on which storage method clients choose (e.g. >4GB for direct deal making and any size for aggregators). While these factors are important considerations, there is no clear 'golden path' for builders and clients to store their data and the feedback is that it makes it that much harder for them to adopt storage on Filecoin via FVM.

Key discussion points:

• Single pieces of client data could potentially have a SBT NFT attached to it that can serve several purposes. Firstly, it complements the authenticator idea mentioned above, where the NFT indicates authentication status and aggregators/SPs can check the status before accepting to store. Secondly, if its a dynamic NFT, it can easily indicate the status of storage and number of replicas on the network, to the client e.g. the storage replication NFT example from @hammertoe. (contributors: @raulk)

• A data wallet is a viable solution here that allows seamless usage of all layers in a single interface. The data wallet functions as an overview of what data the client has stored, storage status of each piece of data and allows easy click-throughs to storage platforms/providers to manage storage. The wallet could be a wallet plug-in or a standalone wallet. It can also incorporate the SBT NFT idea mentioned above. (contributors: @galen-mcandrew @raulk @longfeiWan9 @nandit123 @trruckerfling )

  • Current action: [⚠️ Need community feedback!] FVM product team is gathering feedback from the community if a data wallet is required and what user requirements there are. They aim to provide a standard specs for the wallet, to enable interested ecosystem partners to build from it and provide the service. Dataverse OS is an existing ecosystem project that aims to provide this.

3. What value-added services from aggregators/SPs on the storage side, would incentivize paid storage deals by clients, aside from FIL+?

Context: Currently with FIL+, clients are able to store data for free after completing KYC and receiving 'datacap'. Datacap can be used by clients to fund storage deals with storage providers (SP) and SPs that receive datacap, receive higher quality-adjusted power in the network. For various reasons, SPs observe the need to create more value for clients and subsequently have them partake in paid deals, using FIL and not just datacap.

Key discussion points:

• [⚠️ Need community feedback!] What are some SP-hosted storage services that offer incremental value, to incentivize clients to partake in paid deals? With FVM, builders have the ability to build dApp solutions by manipulating storage primitives on-chain. What do clients require from SPs that would motivate them to pay for storage?
  • Current action: There will be a discussion to get SP feedback on what use case they are keen to build, at the upcoming FIL Vegas, hosted by @hammertoe

4. As programmatic storage markets evolve, what other market components and/or storage workflow building blocks are clients and dApp builders, keen to see? To make PSM participation valuable & appealing?

Context: Since the FVM launch in March 2023, it has revolutionized the programmatic storage market, specifically on storage deal making workflows.

• Firstly, it has enabled proof of data segment inclusion (PoDSI), which allows any size of data to be trackable within storage sectors, rather than being lost within a larger data segment. This is heavily used by aggregators, that aggregate smaller pieces of data before storing it with a SP.
• Secondly, it has enabled direct deal making, where a smart contract can technically make a storage deal with a SP directly. Caveat: there are other considerations from SPs that need to be factored in when making a deal on mainnet.

[⚠️ Need community feedback!] The work has been largely focused on storage deal making workflows. The FDS SG discussion opened up discussions around UX of storage workflows, including data wallets as a solution. Are there other PSM areas that require discussion and/or thought leadership to help it mature? (cc: @aashidham )

[aashidham]

Framing the data programmability conversation

One way to frame this broader discussion on data programmability is to put it into the context of where in the FVM landscape we aim to innovate and scope more deeply.

If we think about the broader FVM project as a series of blocks that we aim to innovate on:

We are really double clicking on the "programmability enhancements" thread. Within this thread, we have programmability verticals like:

1. Defi programmability [which includes wrapped FIL, liquidity pools, built-in actor support] that we have built out in order to grow the Defi landscape
2. Data programmability [which includes upload, aggregation and RaaS flows, as well as PSMs, data wallets and provenance markets]
3. Compute programmability [which includes integration with Lilypad, tooling to grow ML and AI use cases]
4. SP dapp development [tools that can help SPs better do their jobs over time]

The hope is to innovate on #2 with this conversation.

The current landscape of data programmability

We currently have built out a roadmap that includes the development of the aggregator standard and the client contract standard to allow developers two ways to upload and manage storage deals (with an emphasis on the aggregator standard):

Here is where we currently stand with all these buildouts:

	Current Developments	Potential Future Developments
Uploads - CC	- Enables dapps and DAOs to programmatically upload data directly to SPs - CC v1.0 in order to prevent SP frontrunning and to allow 1-1 SP-Client data onboarding - Will likely require offchain communication with SPs - Useful for larger data onboarding	- Building a standard in order to verify client datasets (see #1) so that SPs can onboard more selectively
Uploads - Aggregator Standard	- Enables dapps and DAOs to programmatically upload data to SPs with aggregators - Will NOT require dev to communicate with SPs directly - Can be used with offchain or onchain aggregators (former requires PoDSI, latter does not) - Useful for most smaller data onboarding use cases	- Building a standard in order to group SPs with different characteristics (eg geos, retrieval reputation, callib net support) so that aggregators can choose to work with an SP group - Building a standard in order to verify client datasets (see #1) so that aggregators can onboard more selectively
Raas - CC	- Renewal, replication, and repair of existing storage deals made directly with an SP	- Can SPs offer RaaS in order to incentivize paid storage deals by clients? (see #3)
Raas - Aggregator Standard	- Renewal, replication, and repair of existing aggregated storage deals	- Can aggregators offer RaaS in order to incentivize paid storage deals by clients? (see #3)
Retrievals - CC	- Incenventize the SP to create retrievals on storage deals (with a known SLA for retrievals) - Retrieval oracles can punish SPs if they do not provide timely retrievals	- Retrievals repuation standard may be necessary in order to track SPs with high onchain reputation
Retrievals - Aggregator Standard	- As described above, but proxy through an aggregator service - Aggregators may be able to maintain hot copies of some datasets so not all retrieval requests will have to go directly to SPs	- Build out a retrievals marketplace to do price discovery on the cost of a retrieval job

As you can see, with this structure we can start fitting in our needs for what is next in data programmability in the context of our current roadmap.

The future of data programmability

What falls next falls into three large categories:

• Markets (PSMs, data provenance markets, data preparation markets)
• Standards (SP grouping standard, retrieval reputation standard, SBT in order to carry over deal state)
• Tooling (data wallets, as described in Cleanup #2 above)

You can see this laid out partially in the last column of the table above. Also can be visualized as follows:

The Client <> Aggregator <> SP relationship has special interest to us, since we see lots of builders using this pathway in particular:

Hopefully this framework allows us to iterate on designs and schemas in Iceland with an understanding of the broader context.

[NiloCK]

Commenting on an exchange from early in the discussion:

• SPs have regulatory compliance requirements that force them down trad-kyc, and prohibit wild west web3 permissionless / pseudonymous transactions.
• counterpoint: Maybe in the post FVM world this could be circumvented - can emerging zero-knowledge DiD protocols provide sufficient KYC for requirements while enabling pseudonymous marketplace?
• counterpoint: no matter how sound a DiD standard is, SPs will rely on recognized legal status of these protocols, which is more likely than not to be lagging behind.

But maybe not - in the US at least there are some specific efforts toward knowledge building and adoption of ZK methodologies, at least within the federal government. Maybe that will trickle down?

https://www.whitehouse.gov/wp-content/uploads/2022/01/M-22-09.pdf

[jmazzahacks]

Writing as an app dev who has a dangerously shallow knowledge of how SP's work under the hood, but this is more of idealogical post than anything else.

From the summary -> "If data turns out to be malicious, the aggregator reserves the right to take down the data.", also means -> "If data turns out to be inconvenient for certain political/global actors, the aggregator has the ability to censor that data if compelled." (and plenty other variations of that).

Censorship resistance, it's the raison d'être of web3. Period. So I really hope not to see folks just wring their hands and say this is too hard, we'll worry about it later. re KYC / DataCap / verified deals to on-board a lot of data and bootstrap the network, okay fine, but that can't go on forever.

Respectfully

[wolfduan]

github