new input type for transition tool

[sudeepdino008]

Hello, We at CovalentHQ are developing an indexing solution which ultimately provides REST APIs to read blockchain data. We do this by processing blockchain data, indexing and enriching it, which can be consumed by developers. It's an infra project aimed at providing efficient read access of blockchain data. We’re now looking to progressively decentralize our solution.

As a part of this, we came up with a block data read format called a block specimen which can be defined as "all the data needed by the evm to (re)execute any given block" -- this also includes reads off of storage like SLOAD. This allows us to perform “EVM execution of an arbitrary ethereum block” using just the block specimen binary file, without the need for state storing databases.
Once the block specimen has been created, our decentralized network doesn't have to rely on ethereum nodes, as all data needed to execute the block is captured by the block specimen. The process makes sure that the block specimen data isn’t compromised by multi party cryptographic signing process on the hash of the result in a given time frame.

Resulting block specimens can be run through a data (transformation) pipeline which includes a stateless evm tool (for getting receipts and logs), tracing and annotating, proofing the data etc. It separates out the data storage layer from the block execution and distributed consensus functionality. Here is a sample block specimen json. Here is the block specimen whitepaper for more info.
For the Covalent network, we plan to create a marketplace for such block specimens, wherein the operators that create them (the “block specimen producers” run ethereum nodes with block-specimen-producing patch on top) power the supply side, whereas the refiners/indexers etc. compose the demand side. This part will be crucial in creating a decentralized indexing solution, allowing token holders and developers to engage with the network on their own terms, in new ways.

Our team has worked on importing the evm transition tool from geth (go-ethereum) to erigon (previously turbo-geth). We also have a patch in our fork (covalenthq/erigon), which allows the transition tool (t8n) to accept directly block specimens binary files as inputs, and we thereby make a case for merging this patch into upstream erigon.

Simplifying the input of evm t8n tool

Let's start with the evm t8n tool. The tool currently takes prestate with multiple inputs for example account balances, previous blockhashes, set of txs, mining rewards etc. and outputs a "post state" which has state root, receipts, state dump etc.
Using block specimen here consolidates all the input information into a single input. We can also have as input an entire arbitrary block, and see how the block execution engine behaves while executing that block. It’s a great way to test the execution of an ethereum block, as block specimen is essentially a canonical representation of a block in the EVM ecosystem favoring read operations. It opens up a more simple and concise way of using the transition tool.

block specimen is more comprehensive than current input set

block specimens are more comprehensive than what the current tool input (prestate, set of txs, mining rewards etc.) can capture. One such example is solidity contract state reads. This, similar to the above point, expands the transition tools capabilities.
We’re currently using this block-specimen-accepting transition tool in our network. If you wanna have a look at what the changes look like, check here.
Let us know if the proposal sounds good, or if you have any other concerns. We would love to discuss this with you.

[sudeepdino008]

@AskAlexSharov Hi. What do you think?

[AskAlexSharov]

Hi.

1. i don’t know how to see diffs on forks in Github UI :-) Please teach me or open PR. Then i can take a look on code.
2. try merge latest devel to your fork.
3. FYI: ErigonV3 can produce “read sets” which keys execution did read.
4. FYI: Erigon doesn’t store history as “full states” - it storing history as “inverted diffs” and to exec any block it need only: files with such diffs+some index on them+latest state, to read any historical key/value it need 1 index read and 1 hisory_file_or_latest_state read. ErigonV3 also storing diffs with transaction-granularity (not block granularity) and can store diffs in files outside db. Means can exac any historical txn without executing anything else. And nodes in network will auto-produce such files and serve them by bittorrent.
5. I take a look on your format example and guess: 5.1 main pain will be SELFDESTRUCT :-) 5.2 some things will be not possible - like “generate proofs” (or you will need include part of merkle trie to file) 5.3 some consensuses (bsc, aura, etc..) have own state: what is current validator set, what are last votes result, etc… without consensus rules you may get invalid “block rewards” or something else. Some consensuses may try to read historical Receipts or headers - this is also “hidden state”.

[sudeepdino008]

Hi Alex, thanks for you response. I have some questions...

1. you can see the PR here - https://github.com/covalenthq/erigon/pull/9/files (it's not on latest erigon, but will give you an idea of how things look with block specimen as input)
2. in progress
3. and 4: that's interesting to know.
4. didn't get how SELFDESTRUCT will be a pain point. Can you elaborate?
5. 2 generating proofs is something we're not doing currently, but we'll be adding parts of the merkle trie in order for any executor to be able to validate the state root as well. It is something that's on the roadmap to add in the block specimen format.
6. 3 We'll only be serving the PoS (or PoW) consensus via this (or in general chains which do not include such "hidden state" information). Block specimen is intended to be for EVM compatible chains, and I guess that such modified consensuses with differing block rewards etc. deviate from the yellow paper specification.

[AskAlexSharov]

4. selfdestruct - it's when: in 1 block happens "create new storage key S1", "delete account with all storage", "create account with same address and empty storage", "create new storage record S2".
5. We now have example of eth_getProof: Add eth_getProof support for historical blocks #7115

[sudeepdino008]

4. The block specimen represents a state prior to block execution, once it is loaded into the "structures which provide state data during block execution", those structure should be able to handle edits, including reading slots which are updated during block execution, or selfdestruct scenario where a new account is created on a selfdestruct-ed address.
   I feel like I'm missing something here. Is it that currently this is not supported, and won't work?
5. ACK.

The point is to transition some of the read scalability/availability challenges away from ethereum nodes. In the sense that, Vitalik said about the Purge - “Clients will stop storing history that’s more than 1 year old and we shift the responsibility of storing and retrieving that history to other protocols.”
This responsibility goes beyond storing data in archived cold storage, and involves enriching, indexing, and even proofing.

[AskAlexSharov]

True. ErigonV2 already did move some historical data (with enrich and indexing) to Bittorrent protocol. ErigonV3 does move more such data (with more enrichment and indexing) to Bittorent protocol. It can be proofed (by state root, and other hashes). ErigonV4 will move history of commitments (proofs) there also.

We don't plan to introduce new protocol for history, but may adopt something from "Vitalik" in future.