IPLD Crete Resolutions

[nicola]

@Stebalien, @dignifiedquire and I had a set of conversations on:

• IPLD types
• IPLD changes in the spec ({'/': '...', type:''})
• IPLD selector

@Stebalien can you write down the notes that we took?
I don't have the notes with me, I think they are on your notebook

[Stebalien]

They are. I'll write them up later today.

[daviddias]

@Stebalien Looking forward to reading your write up. I've had a good chat with @nicola and we identified some misconceptions about how IPLD should work. There are simplifications that we should continue pursuing dev exp and maintenance, example: JSON doesn't have Buffers, that is fine, we don't need to recreate EJSON/BSON + JavaScript Objects are not JSON.

[nicola]

@Stebalien here in Lisbon we kind of reverted the resolution to the second proposal,
if a format can't support a type, an IPLD object that has that special type can't convert back to it (for example, JSON)

[dignifiedquire]

@nicola why would you do that? That leaves with a much worse solution than what we had at the end of our talks in Crete.

[Stebalien]

Honestly, we should just avoid JSON as much as possible. However, JSON is useful for interoperability with, e.g., tools like jq. Not being able to query over most IPLD data with tools like jq would be unfortunate.

[nicola]

@diasdavid, this is the time in which you explain your reasoning :D

[daviddias]

We want to enable users to:

• Store JSON directly and retrieve it with 1:1 mapping
• Store JavaScript Objects and retrieve them with 1:1 mapping. JavaScript Objects are not JSON. JavaScript Objects can be stored with 1:1 mapping to and back from CBOR.
• Store Go Structs and retrieve them with 1:1 mapping. Go Structs are also not JSON and they do also have 1:1 mapping with CBOR.

We don't want to:

• Create a new type of JSON. If we wanted, we could have used EJSON/BSON from the start which would add all of this stuff underneath with magic for users.

We also want:

• Help users make good decisions. Serializing a JavaScript Object that has a Buffer to a dag-json format should error with a message "dag-json doesn't have any way to represent Buffers".

As a side note to the side quest, the real issue with JSON and why we keep hitting the JSON wall is because it gets used as the data format to create responses on the HTTP-API. However, me, @whyrusleeping and I believe also @Stebalien agree that defaulting to JSON on the http-api is just a cause of great pain and not any gain. We want the users to pick however and the http-api should return a header with the serialization type (i.e application/dag-cbor). The way to solve the http-api situation is described here https://github.com/ipfs/interface-ipfs-core/issues/81#issuecomment-277271941

[Stebalien]

Couple of counter points:

1. We've already introduced a new type of JSON, DagJSON (we have special ways to write CIDs).
2. As I said above, lots of tools expect JSON-like formats and refuse to operate over anything else.
3. JSON-like formats are human readable.

agree that defaulting to JSON on the http-api is just a cause of great pain and not any gain.

I do agree however, the real cause of pain there is the fact that pure JSON can't represent byte arrays (precisely what we were trying to address). I do believe we should stop returning JSON from the HTTP endpoint by default, for performance if nothing else. However, being able to spit out JSON (or something like it) on the command line is still quite useful, if for no other reason than human readability (*without loss of important information as is the case today).

Now, we could just add support for EJSON, however, IMO, it isn't the best fit:

1. We don't really want to support dates (do we?).
2. Reserves everything starting with a $.
3. Doesn't use multibase (ok, maybe not that important but this would still be a "nice to have").

---

TL;DR:

1. We need some format to display data that doesn't confuse users by throwing away important information (like, "this is binary").
2. It would be nice to be able to use tools like jq on arbitrary IPLD objects.

[rvagg]

Closing due to staleness as per team agreement to clean up the issue tracker a bit (ipld/team-mgmt#28). This doesn't mean this issue is off the table entirely, it's just not on the current active stack but may be revisited in the near future. If you feel there is something pertinent here, please speak up, reopen, or open a new issue. [/boilerplate]