[DOC] Document new changes RPC API

[frol]

UPDATE: I completely rewrote this description!

As of near/nearcore#2148 we have a brand new API for the changes endpoint. We expose new APIs with a prefix EXPERIMENTAL_, so currently one can run a local node and use EXPERIMENTAL_changes endpoint to get the changes that are happening on the network on a per-block basis.

UPDATE 2020/03/16: There is an upcoming update near/nearcore#2264, which will replace account_id in the request with account_ids expecting a list of account_id (I chose to make it a breaking change while we are at the experimental stage).

NOTE: There is an extended EXPERIMENTAL_changes_in_block API: #267

API

EXPERIMENTAL_changes RPC endpoint accepts the following types of requests:

{
    "block_id": /* block hash | block height | omit it if you want to use `finality` */,
    "finality": /* "latest" | "final" | "near-final" | omit it if you want to se `block_id` */,
    "changes_type": /* "account_changes" | "single_access_key_changes" | "all_access_key_changes" | "contract_code_changes" | "data_changes" */,
    /* extra fields based on `changes_type` */
}

the extra fields based on changes_type are the following:

• "account_changes":
  • account_ids as list of account_id
• "single_access_key_changes":
  • keys as list of objects:
    • account_id
    • public_key
• "all_access_key_changes":
  • account_ids as list of account_id
• "contract_code_changes:
  • account_ids as list of account_id
• "data_changes":
  • account_ids as list of account_id
  • key_prefix_base64 (base64-encoded string of the key-prefix data saved from a smart contract)

Example of Account changes

When an account gets modified (created, updated, removed), we can query the block which contains the receipts for the transaction:

http post http://localhost:3030 jsonrpc=2.0 id=dontcare \
    method=EXPERIMENTAL_changes \
    'params:={
        "block_id": "DYHRzRbxUR1ANPdCQcgQE9g5zyYQoDZ1k8BJEQ3hDSgW"
        "changes_type": "account_changes",
        "account_ids": ["new-account"]
    }'

we observe the following:

{
    "block_hash": "DYHRzRbxUR1ANPdCQcgQE9g5zyYQoDZ1k8BJEQ3hDSgW",
    "changes": [
        {
            "cause": {
                "type": "receipt_processing",
                "receipt_hash": "8rPN9tHY9MtT23TqBvvBGBQzwskA5fuZDjmiLpTobgRv"
            },
            "type": "account_update",
            "change": {
                "account_id": "new-account",
                "amount": "0",
                "locked": "0",
                "code_hash": "11111111111111111111111111111111",
                "storage_usage": 100,
                "storage_paid_at": 9
            }
        }
    ]
}

If the account was deleted:

{
    "block_hash": "DYHRzRbxUR1ANPdCQcgQE9g5zyYQoDZ1k8BJEQ3hDSgW",
    "changes": [
        {
            "cause": {
                "type": "receipt_processing",
                "receipt_hash": "8rPN9tHY9MtT23TqBvvBGBQzwskA5fuZDjmiLpTobgRv"
            },
            "type": "account_deletion",
            "change": {
                "account_id": "new-account"
            }
        }
    ]
}

Example of Data changes

When some key gets modified (created, updated, removed), we can query the block which contains the receipts for the transaction:

http post http://localhost:3030 jsonrpc=2.0 id=dontcare \
    method=EXPERIMENTAL_changes \
    'params:={
        "block_id": "DYHRzRbxUR1ANPdCQcgQE9g5zyYQoDZ1k8BJEQ3hDSgW"
        "changes_type": "data_changes",
        "account_ids": ["new-account"],
        "key_prefix_base64": "bXlfa2V5"  /* base64-encoded "my_key" */
    }'

we observe the following (assume the key was modified twice in a single block, and then removed):

{
    "block_hash": "DYHRzRbxUR1ANPdCQcgQE9g5zyYQoDZ1k8BJEQ3hDSgW",
    "changes": [
        {
            "cause": {
                "type": "receipt_processing",
                "receipt_hash": "111N9tHY9MtT23TqBvvBGBQzwskA5fuZDjmiLpTob111",
            },
            "type": "data_update",
            "change": {
                "account_id": "new-account",
                "key_base64": "bXlfa2V5",  /* base64-encoded "my_key" */
                "value_base64":  "bXlfdmFsdWVfMQ==" /* base64-encoded "my_value_1" */,
            }
        },
        {
            "cause": {
                "type": "receipt_processing",
                "receipt_hash": "222N9tHY9MtT23TqBvvBGBQzwskA5fuZDjmiLpTob222"
            },
            "type": "data_update",
            "change": {
                "account_id": "new-account",
                "key_base64": "bXlfa2V5",  /* base64-encoded "my_key" */
                "value_base64": "bXlfdmFsdWVfMg==" /* base64-encoded "my_value_2" */
            }
        },
        {
            "cause": {
                "type": "receipt_processing",
                "receipt_hash": "333N9tHY9MtT23TqBvvBGBQzwskA5fuZDjmiLpTob333"
            },
            "type": "data_deletion",
            "change": {
                "account_id": "new-account",
                "key_base64": "bXlfa2V5"  /* base64-encoded "my_key" */
            }
        }
    ]
}

Notes

• While the above examples seem to be accurate, there is a better source of truth -- the tests: https://github.com/nearprotocol/nearcore/blob/60bd4ab09880cbd0a8027dbaf713757cdb718aff/pytest/tests/sanity/rpc_key_value_changes.py
• This API is yet to be exposed in near-api-js: Expose changes RPC API near-api-js#196

[vgrichina]

What errors are possible / how they will be returned?

[vgrichina]

key_prefix_as_array_of_bytes are we going to end this madness at some point? If array of bytes needs to be sent over JSON, it should be base-encoded.

In this particular case it also would make sense to accept UTF-8 strings.

[vgrichina]

Also it really looks like this should have been REST API:

/account/{account_id}/{changes_type}/{key_prefix|data_id|access_key_pk}

If we make API easy to use directly we won't have to maintain special wrapper library for every language.

[frol]

key_prefix_as_array_of_bytes are we going to end this madness at some point? If array of bytes needs to be sent over JSON, it should be base-encoded.

Good point! I have created the tracking issue for this: near/nearcore#2034

In this particular case it also would make sense to accept UTF-8 strings.

The keys are not required to be UTF-8 as far as I know, and given base64 is a string, it will be confusing.

Also it really looks like this should have been REST API

What is the point of introducing REST over RPC? I am, actually, redesigning the API (started with query, which uses account/<account_id> encoding, which does not really help and only makes things hacky).

[frol]

@amgando I had updated the description to reflect the recent API redesign.

Just for the reference:

• Initial implementation was here: Queryable state changes nearcore#1910 (mosty irrelevant now)
• New implementation is here: feat(chain): Improved changes RPC nearcore#2148
• The issues that trigger the new implementation:
  • Update changes RPC to operate with base64-encoded data instead of arrays of bytes nearcore#2034
  • Failed to query account changes nearcore#2048

[frol]

I had updated the issue to reflect the upcoming changes in near/nearcore#2264. account_id in the request is going to be renamed into account_ids and will expect a list of account_id (I chose to make a breaking change while we are at the experimental stage).