Add information byte to message share format

[musalbas]

The current share format is as follows:

nid (8 bytes) | message length (varint) | share data for shares at the start of a message;
nid (8 bytes) | share data for other shares.

I propose to change it to the following:

nid (8 bytes) | info byte | message length (varint) | share data for shares at the start of a message;
nid (8 bytes) | info byte | share data for other shares.

Where info byte is a byte with the following structure:

• the first 7 bits are reserved for the version information in big endian form (initially, this will just be 0000000 until further notice);
• the last bit is a "message start indicator", that is 1 if the share is at the start of a message.

Rationale:

1. The message start indicators allow clients to parse a whole message in the middle of a namespace, without needing to read the whole namespace.
2. The version bits allow us to upgrade the share format in the future, if we need to do so in such a way that different share formats can be mixed within a block.

[rootulp]

Thanks for the visual depictions of the current/proposed share format, they really help!

The message start indicators allow clients to parse a whole message in the middle of a namespace, without needing to read the whole namespace.

Clarifying question: is this because a a client cannot differentiate between message length (varint) and arbitrary share data in the current format?

Does the proposed format include the message start indicator in the info byte (as opposed to a separate byte) to save one byte per share? This seems like a sane tradeoff to make assuming we don't expect to exhaust 2^7 versions.

[musalbas]

Yes to both. Also, we could use the info byte for other types of signals in the future, if we need to.

…

On 18 August 2022 20:17:19 Rootul Patel ***@***.***> wrote: Thanks for the visual depictions of the current/proposed share format, they really help! > The message start indicators allow clients to parse a whole message in the > middle of a namespace, without needing to read the whole namespace. Clarifying question: is this because a a client cannot differentiate between `message length (varint)` and arbitrary `share data` in the current format? Does the proposed format include the message start indicator in the info byte (as opposed to a separate byte) to save one byte per share? This seems like a sane tradeoff to make assuming we don't expect to exhaust 2^7 versions. -- Reply to this email directly or view it on GitHub: #839 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

[adlerjohn]

assuming we don't expect to exhaust 2^7 versions.

The info byte could always be extended, e.g. if the 7 bits are 1111111 then read the second byte.

The proposal seems fine, I have no objections. Adding versions to things has proven to basically always be a good thing in hindsight. The overhead is minimal, especially considering we may increase share size.

[rootulp]

Closing in favor of celestiaorg/celestia-app#659