feature(chain): Exposed accounts touched within the block #2253

frol · 2020-03-13T04:24:56Z

Resolves #2128

API:

$ http post http://localhost:3030 jsonrpc=2.0 id=dontcare \
    method=EXPERIMENTAL_changes_in_block \
    'params:={"block_id": "DYHRzRbxUR1ANPdCQcgQE9g5zyYQoDZ1k8BJEQ3hDSgW"}'

$ http post http://localhost:3030 jsonrpc=2.0 id=dontcare \
    method=EXPERIMENTAL_changes_in_block \
    'params:={"block_id": 100500}'

$ http post http://localhost:3030 jsonrpc=2.0 id=dontcare \
    method=EXPERIMENTAL_changes_in_block \
    'params:={"finality": "latest"}'

NOTE: The endpoint naming needs some bikeshedding.

The response:

{
    "block_hash": "DYHRzRbxUR1ANPdCQcgQE9g5zyYQoDZ1k8BJEQ3hDSgW",
    "changes": [
        {
            "type": "data_touched",
            "account_id": "frol.near"
        },
        {
            "type": "account_touched",
            "account_id": "frol.near"
        },
        {
            "type": "access_key_touched",
            "account_id": "user.near"
        },
    ]
}

gitpod-io · 2020-03-13T04:25:01Z

frol

I am not absolutely happy with the naming, so I am open to suggestions.

frol · 2020-03-13T04:25:59Z

chain/chain/src/store.rs

+    /// Retrieve the kinds of state changes occurred in a given block.
+    ///
+    /// We store different types of data, so we prefer to only expose minimal information about the
+    /// changes (i.e. a kind of the change and an account id).
+    fn get_key_value_changes_in_block(
+        &self,
+        block_hash: &CryptoHash,
+    ) -> Result<StateChangesKinds, Error> {


FYI, This is the core part of the PR.

core/primitives/src/utils.rs

pytest/tests/sanity/rpc_key_value_changes.py

core/primitives/src/types.rs

core/primitives/src/utils.rs

evgenykuzyakov · 2020-03-14T02:16:44Z

@frol  It would also be ideal to not base API for any particular state implementation. E.g. our current serialization with `col` can change later, e.g. if we find it beneficial to build a merkle tree per account. Or some other implementation. 

What is not going to change is API's results, e.g. the format in which you return results already. So it would be good to use input filter to be similar based on `account_id` or suffix/prefix of a given `account_id` (if needed) instead of raw keys.

It's already not the case. Nevermind :)

…e stage to be compatible with the upcoming hashed account_id design

core/store/src/trie/mod.rs

frol · 2020-03-14T05:28:03Z

@evgenykuzyakov @nearmax I need your help.

This latest commit failed on a test, and this raised a question I had before. Do we want to expose the following types of changes via changes API?

    pub const POSTPONED_RECEIPT_ID: &[u8] = &[4];
    pub const PENDING_DATA_COUNT: &[u8] = &[5];
    pub const POSTPONED_RECEIPT: &[u8] = &[6];
    pub const DELAYED_RECEIPT_INDICES: &[u8] = &[7];
    pub const DELAYED_RECEIPT: &[u8] = &[8];

At least DELAYED_RECEIPT_INDICES gets saved into Trie (this is why the test fail, as I have not implemented key parsing for it, though, there is no info in the key, the whole key is just [7], which makes me believe that it is a "singleton")

core/store/src/trie/mod.rs

evgenykuzyakov · 2020-03-15T23:54:11Z

At least DELAYED_RECEIPT_INDICES gets saved into Trie (this is why the test fail, as I have not implemented key parsing for it, though, there is no info in the key, the whole key is just [7], which makes me believe that it is a "singleton")

It's a singleton per shard. It keeps the indices of the delayed receipts queue:
https://github.com/nearprotocol/nearcore/blob/175f42076fb383ffcfc565942e43e53183f09c01/runtime/runtime/src/lib.rs#L1058-L1060

pub const RECEIVED_DATA: &[u8] = &[3]; - Data received for a key data_id. The required postponed receipt might be still not received or requires more pending input data.

pub const POSTPONED_RECEIPT_ID: &[u8] = &[4]; - receipt ID for a key data_id. The Data is not available and is needed for the postponed receipt to execute.

pub const PENDING_DATA_COUNT: &[u8] = &[5]; - the number of missing data inputs that are still not available for a key receipt_id.

pub const POSTPONED_RECEIPT: &[u8] = &[6]; - the receipt itself for a key receipt_id.

pub const DELAYED_RECEIPT_INDICES: &[u8] = &[7]; - the singleton described above

pub const DELAYED_RECEIPT: &[u8] = &[8]; - delayed receipt, because the shard is overwhelmed.

frol · 2020-03-16T01:28:14Z

@evgenykuzyakov Thank you for the information! I will put your comments into the code. It seems that that info is only relevant for internal operation, and given that we don't expose it via API yet, I can skip saving those changes. Though, reading all the above I realized that we are missing shard information in our APIs. It seems that we need to pass shard_id into WrappedTrieChanges and attach it to the stored value and expose it via RPC. Let me know if that does not make sense to you.

…se them via RPC anyway

…e/2128-RPC-expose-changes-kinds-per-block

evgenykuzyakov · 2020-03-16T16:43:23Z

Though, reading all the above I realized that we are missing shard information in our APIs. It seems that we need to pass shard_id into WrappedTrieChanges and attach it to the stored value and expose it via RPC.

I'm not sure you need to do this, because shard_id will be always extracted from a pair of AccountId and BlockHeight. So shard_id is more an internal concept. We should not expose it to the RPC if we have an AccountID.

bowenwang1996 · 2020-03-16T18:48:13Z

Yeah we should not expose shard_id to rpc because it should not be external in any way (it can be subject to change internally).

… KeyForStateChanges

…e/2128-RPC-expose-changes-kinds-per-block

… nightly*.txt

frol · 2020-03-17T00:40:41Z

I believe this PR is ready to be merged.

frol · 2020-03-17T01:27:30Z

The follow-up documentation request is near/docs#267

Introducing `TrieKey` trait to work with keys within `TrieUpdate` instead of using raw `Vec<u8>`. This allows to have additional logic for keys. E.g. if we want to hash AccountID for keys (see #2215) and later return all touched accounts for a block (see #2253) we need a way to map `sha256(account_id)` from a raw key back to the `account_id`. Then `TrieKey` may implement `fn get_account_id_with_hash() -> Option<(AccountId, CryptoHash)>` logic that provides Account ID and hash of this Account ID. With this info we can remember pairs of touched Accounts in memory of `TrieUpdate` and now we can map back hashes here: https://github.com/nearprotocol/nearcore/blob/62f00b211915790d06e822da866fb01ceb6cdc89/core/store/src/trie/mod.rs#L578 Removes iterators from VMLogic (cause they somewhat block changes from this PR). Fixes #2276 ##Test Plan: - unit tests and CI

feature(chain): Exposed accounts touched within the block

8a70115

frol requested review from bowenwang1996, evgenykuzyakov, MaksymZavershynskyi and SkidanovAlex as code owners March 13, 2020 04:24

frol commented Mar 13, 2020

View reviewed changes

evgenykuzyakov reviewed Mar 13, 2020

View reviewed changes

core/primitives/src/types.rs Outdated Show resolved Hide resolved

core/primitives/src/utils.rs Show resolved Hide resolved

evgenykuzyakov approved these changes Mar 13, 2020

View reviewed changes

pushed StateChangeKind deduction from the read stage down to the writ…

1a5cde3

…e stage to be compatible with the upcoming hashed account_id design

frol commented Mar 14, 2020

View reviewed changes

core/store/src/trie/mod.rs Show resolved Hide resolved

frol requested a review from evgenykuzyakov March 14, 2020 05:16

frol commented Mar 14, 2020

View reviewed changes

core/store/src/trie/mod.rs Show resolved Hide resolved

frol added 2 commits March 16, 2020 03:00

fixed the broken test by skipping all the unknown state we don't expo…

97b724c

…se them via RPC anyway

Merge branch 'master' of github.com:nearprotocol/nearcore into featur…

ddfd169

…e/2128-RPC-expose-changes-kinds-per-block

frol mentioned this pull request Mar 16, 2020

GC major update #2084

Merged

frol added 3 commits March 16, 2020 21:20

refactored an ad-hoc creation of a storage key for state changes into…

cc83097

… KeyForStateChanges

Merge branch 'master' of github.com:nearprotocol/nearcore into featur…

62cd912

…e/2128-RPC-expose-changes-kinds-per-block

fixed the renamed rpc_key_value_changes.py to rpc_state_changes.py in…

e776d5d

… nightly*.txt

frol assigned evgenykuzyakov and bowenwang1996 and unassigned evgenykuzyakov Mar 17, 2020

bowenwang1996 approved these changes Mar 17, 2020

View reviewed changes

frol merged commit 76996c1 into master Mar 17, 2020

This was referenced Mar 17, 2020

[DRAFT] Hash account_id. Update StateRecord #2244

Closed

[DOC] Document new EXPERIMENTAL_changes_in_block RPC API near/docs#267

Closed

frol mentioned this pull request Mar 17, 2020

expose balance-change events in a block #2036

Closed

evgenykuzyakov mentioned this pull request Mar 19, 2020

feat: Switch TrieUpdate API to use TrieKey trait #2278

Merged

frol deleted the feature/2128-RPC-expose-changes-kinds-per-block branch March 20, 2020 14:51

frol mentioned this pull request Mar 25, 2020

feat: Refactor TrieKey #2318

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature(chain): Exposed accounts touched within the block #2253

feature(chain): Exposed accounts touched within the block #2253

frol commented Mar 13, 2020

gitpod-io bot commented Mar 13, 2020 •

edited

Loading

frol left a comment

frol Mar 13, 2020

evgenykuzyakov commented Mar 14, 2020 •

edited

Loading

frol commented Mar 14, 2020

evgenykuzyakov commented Mar 15, 2020

frol commented Mar 16, 2020

evgenykuzyakov commented Mar 16, 2020

bowenwang1996 commented Mar 16, 2020

frol commented Mar 17, 2020

frol commented Mar 17, 2020

feature(chain): Exposed accounts touched within the block #2253

feature(chain): Exposed accounts touched within the block #2253

Conversation

frol commented Mar 13, 2020

gitpod-io bot commented Mar 13, 2020 • edited Loading

frol left a comment

Choose a reason for hiding this comment

frol Mar 13, 2020

Choose a reason for hiding this comment

evgenykuzyakov commented Mar 14, 2020 • edited Loading

frol commented Mar 14, 2020

evgenykuzyakov commented Mar 15, 2020

frol commented Mar 16, 2020

evgenykuzyakov commented Mar 16, 2020

bowenwang1996 commented Mar 16, 2020

frol commented Mar 17, 2020

frol commented Mar 17, 2020

gitpod-io bot commented Mar 13, 2020 •

edited

Loading

evgenykuzyakov commented Mar 14, 2020 •

edited

Loading