Market Data API for direct data onboarding

[snissn]

User Story

People want visibility into how data and deals are being stored on filecoin. We want an API to provide information about direct fil+ deals.

Direct FIL+ is a label for changes to the Filecoin built-in actors, and the components that interact with them, to enable gas-cheap data onboarding, including Filecoin Plus. The built-in storage market actor is bypassed, and the verified registry’s on-chain allocation/claim record suffices to represent a simple “deal”. Direct data onboarding can’t handle client payments (yet), but most deals today have no on-chain payment anyway.

After the initial Direct FIL+ changes, subsequent, smaller changes will be possible to support scalable FIL+ allocations (single transaction covering multiple sectors) and flexibility for other kinds of data applications.

for more see direct data onboarding notion doc https://www.notion.so/pl-strflt/Direct-FIL-impacts-work-outline-57399e0fdfd242d09aabad7357945798

Acceptance Criteria

Deliverables

Preview Give feedback

1. a new api that can get all of the data pieces information from all of the sectors either the data is onboarded via f05 or not.

Technical Breakdown

Development

Preview Give feedback

1. find the types need to make a migration (SectorDealIDs vec)
2. Expose API.
3. Include data sectors either the data is onboarded via f05 or not
4. return provider
5. return client
6. return piece cid
7. return piece size
8. return - market address - f05

Testing

Preview Give feedback

1. Query API and validate results

[jennijuju]

the market address could be null too.

might worth showing if the data has associated datacap as well?

[jennijuju]

Should we show data availability time too? i.e: when its stated to be stored , and till when? (i assume that would could just be the sector lifetime?

[anorth]

There are three pieces of data that will become independent, but are currently coupled.

• A data piece and its commitment into a sector
• A verified claim for a piece (that grants a power multiplier)
• A deal with the built-in market actor (that might involve payments)

A data piece might be accompanied by none, either, or both the other two.

In the future, there will also be

• A "deal" with any other user-programmed actor that functions as a market

...though Lotus shouldn't be expected to interpret state of user-programmed actors, merely provide generic access to it.

a new api that can get all of the data pieces information from all of the sectors either the data is onboarded via f05 or not.

As written, this deliverable must be rooted in the raw piece information, though you should think carefully whether this is the right deliverable to ask for.

Piece commitments will not generally be available in chain state, only in the message history. Thus to provide this information, Lotus needs to index the piece commitments as they appear in the new ProveCommit2/ProveReplicaUpdate3 messages to be developed. For sectors that are onboarded via the legacy methods, the piece commitments are present in the corresponding built-in market deal state (until those deals expire, which is independent of the sector expiration).

Given a piece CID/size, Lotus could then index both the market deal state and verified registry claims by piece+sector in order to associate that related information, if it exists.

It's worth noting that a more restricted deliverable would be much easier to support. If scope were limited to insight into only verified pieces, or only deals, then indexing message history would be unnecessary, and the relevant information would all be available in chain state.

[jennijuju]

okay.. the more I think about this the more I believe lotus is not the right place to provide a piece <> sector indexing, I would imagine this to be something covered along with the other data dumps lily has https://lilium.sh/data/dumps/

Spec-Ed out in Filecoin data piece <> sector indexers/dumps here

[rjan90]

Superceeded by #11614