Add status endpoint to check pin and deal info per cid

[olizilla]

GET /user/uploads/:cid

Proposed respsonse shape:

{
  "cid": "bafy",
  "dagSize": 101,
  "pins": [{
    "peerId": "12D3KooWR1Js",
    "peerName": "who?",
    "region": "where?",
    "status": "Pinned"
  }],
  "deals": [{
    "dealId": 12345,
    "miner": "f99",
    "status": "active",
    "activation": "<iso timestamp>",
    "pieceCid":  "baga",
    "dataCid":  "bafy",
    "dataModelSelector": "Links/0/Links"
  }]
}

This is the subset of properties in the current w3 schema that have some relevance to the user, flattened. Most Batch info is removed, apart from the pieceCid, the dataCid (the cid of the batch) the dataModelSelector, which, I'm told, could be used to extract the cid from the Piece.

The full shape of the schema that is available can be seen in the Fauna GraphQL explorer below for comparison

[olizilla]

@ribasushi is dataModelSelector the right name for it for a user facing api? Does it define the "path through the piece to extract your file"? dataSelector might be better or piecePath could pair nicely with pieceCid... but I'm not clear enough on how it's used to say. pls advise!

[olizilla]

also, i'd be reaching for createdAt in place of activation... or if activation is a term of art here, then perhaps activatedAt?

[olizilla]

As for the enpoint path /user/uploads is already used to list and delete a users content, so we could go with GET /user/uploads/:cid/status or similar to be consistent with "you can uses this to get info on cids you provided*, as opposed to a more general purpose cid checking api that would let anyone check the status of any cid. I'm assuming we won't provide that here. Please confirm @jnthnvctr

[ribasushi]

is dataModelSelector the right name for it for a user facing api?

As far as I am concerned - 100%, and this is what the corresponding lotus CLI/API part is called filecoin-project/lotus#6393 (comment). Caveat emptor: opinions differ on this somewhat

i'd be reaching for createdAt in place of activation... or if activation is a term of art here, then perhaps activatedAt?

I'd use activation yeah. In corner cases it can be weeks between a specific piece-cid<=>miner deal being "conceived" and it actually landing on chain.

[olizilla]

@ribasushi thanks! Should we include the batchCID here too? is it the batch cid or the piece cid that one would need to apply the dataModelSelector ?

[ribasushi]

Errrr wait... I didn't read this close enough :(

• Batch should go from user-facing stuffs: I do not refer to anything bybatch anymore, it just remained in my schema, and @alanshaw picked it up
• The correct term in filcoin-land is Data Cid or Root Cid, take your pick
  • Users absolutely need that, no retrieval is possible without it
  • It usually starts with baf.... ( it's a typical IPFS-ey thing )
• The piece cid is not something that actually plays a role during retrieval, it's what you use if you want to see the status of something on chain yourself ( a stretch for vast majority of users )
  • The PieceCid encoding is always baga...

[olizilla]

And that Data Cid Is what we (internally) store as the Batch cid, right?

[mikeal]

love it!

[alanshaw]

I think GET /user/uploads/:cid/status is good... if it's scoped to the user's uploads.

Apart from the missing batchCid/dataCid/rootCid the response looks great.

If this is authenticated then we can also include the upload name (and created date?), which might be helpful.

I originally thought we'd only have a public status API but I'm struggling to think of use cases where this is beneficial - help?

[jnthnvctr]

I think the benefit to a public status api is contingent on what you’re building - no? it feels like especially since web3 is supposed to be open we might run into cases where this comes up more frequently - valist as an example may want to act as the “storer” for their users packages, but wouldn’t necessarily want each relay mode to need to proxy through them to check on the deal info / status of a cid right? Like in the ideal version this info would not be maintained by us but queryable out of a public deal index - so maybe better form to map to that? I guess my reaction is slightly the reverse - where if these are all stored on a public network wouldn’t it make more sense for the status api to be public? I think making this api scoped to users might (slightly further) remove of us from being permissionless

…

On Tue, Jul 13, 2021 at 6:52 PM Alan Shaw ***@***.***> wrote: I think GET /user/uploads/:cid/status is good... if it's scoped to the user's uploads. Apart from the missing batchCid/dataCid/rootCid the response looks great. If this is authenticated then we can also include the upload name (and created date?), which might be helpful. I originally thought we'd only have a *public* status API but I'm struggling to think of use cases where this is beneficial - help? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#78 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFAHCS3QNNWJJNSMQKKC6LLTXS7UDANCNFSM5AKADQ7A> .

[ribasushi]

And that Data Cid Is what we (internally) store as the Batch cid, right?

@olizilla correct!

[olizilla]

@jnthnvctr i'm inclinded to start with "a user can ask about the status of CIDs they have stored". It would then not be much work to open that api to allow anyone to check the status of a CID.

[olizilla]

@ribasushi just for my context, when you said

Batch should go from user-facing stuffs: I do not refer to anything by batch anymore

...what do you refer to a batch as now?

[ribasushi]

@olizilla I call it aggregate. It can simply be called deal as well. batch implies a certain set of queuing properties we do not actually do, is why I want to move away from it.

[olizilla]

For the scope of this enpoint we could

1. Require authentication (logged in user) and authorization (you are the user that uploaded the CID)
2. Require authentication, but any logged in user can check any CID
3. Require nothing. No auth, No masters. TOTAL FREEE3DOOM. anyone can ask us the status of any CID.

I assumed I was building 1. where a user could check on their stuff. But it is simpler to build the public version. The concern there would be folks hammering the api, and a nebulous concern about leaking info about what our users are storing. Filecoin deal info is public, but I don't think it's currently possible for anyone to determine which CIDs are contained within which deals. On the flip side, you'd also have to already have the content or be given the CID to ask about it, so this may not be a concern.

[ribasushi]

Require nothing. No auth, No masters. TOTAL FREEE3DOOM. anyone can ask us the status of any CID.

The above is 100% the desired endstate, be it M1 or M2.

[ribasushi]

I don't think it's currently possible for anyone to determine which CIDs are contained within which deals

@olizilla every deal our aggregator makes carries a complete manifest of what is contained therein, as the first entry of the first directory, by design. This is to allow indexers and plain curious bystanders to attempt a retrieval of ... --datamodel-path-selector 'Links/0/Hash' and examine the few MiB of content without needing to retrieve the entire thing.

[jnthnvctr]

Agreed with @ribasushi I think we want to do (3) - but this is a good reminder that in the docs (cc @terichadbourne ) we should be heavily emphasizing that these are public open networks and the onus is on users to be encrypting before sticking into these systems.

@olizilla re: your last comment - I think the last bit is right, for a lot of these concerns they require knowledge about a CID already (if they had the CID they could also just request teh content out of IPFS), so its not necessarily leaking additional information. In the end state of what a deal indexing should surface, you would be able to come with a CID to request deal info (which miners have X CID) - so I'd nudge us to mirror that (eventual) world.

I think that does mean that the info contained in the Status API should only be the public network info (so not exposing user defined names, creation time) - as even in the end state of a public deal index we wouldn't have those.

[olizilla]

@jnthnvctr GOOD NEWS. I read the vibe. You and everyone commented in favour of option 3, so that is what I have implemented in #82

[mikeal]

Is it possible to get the timestamp of when the CID was first written to web3.storage and when each deal completed?

we want metrics on this anyway, so we want to make sure we capture all that data. if we have the data we might as well expose it here too.

[olizilla]

Yes! we have a created property on the content object in the schema. I shall add it to the response.