itests: validate responses from ethereum json-rpc api

[iand]

Related Issues

Proposed Changes

Validate responses from following ethereum api calls and fix problems detected (see below for summary of fixes)

• eth_accounts
• eth_blockNumber
• eth_call ✅ fixed in release/1.20.0 branch
• eth_chainId
• eth_estimateGas
• eth_feeHistory ✅ fixes applied
• eth_gasPrice
• eth_getBalance
• eth_getBlockByHash ❌ fevm: EthTx serialization does not match openrpc spec #10106
• eth_getBlockByNumber ❌ fevm: EthTx serialization does not match openrpc spec #10106
• eth_getBlockTransactionCountByHash
• eth_getBlockTransactionCountByNumber
• eth_getCode
• eth_getFilterChanges ✅ fixed in release/1.20.0 branch
• eth_getFilterLogs ✅ fixed in release/1.20.0 branch
• eth_getLogs ✅ fixed in release/1.20.0 branch
• eth_getStorageAt ✅ fixed in release/1.20.0 branch
• eth_getTransactionByBlockHashAndIndex ❌ fevm: EthTx serialization does not match openrpc spec #10106
• eth_getTransactionByBlockNumberAndIndex ❌ fevm: EthTx serialization does not match openrpc spec #10106
• eth_getTransactionByHash ❌ fevm: EthTx serialization does not match openrpc spec #10106
• eth_getTransactionCount
• eth_getTransactionReceipt ✅ fixes applied
• eth_maxPriorityFeePerGas
• eth_newBlockFilter ✅ fixes applied
• eth_newFilter
• eth_newPendingTransactionFilter ✅ fixes applied
• eth_sendRawTransaction
• eth_uninstallFilter

Changes made to fix conformance:

• Added json marshal/unmarshal methods to EthFilterID and EthSubscriptionID
• Changed EthFeeHistory.OldestBlock from uint64 to EthUint64
• Ensured event log topic parsing was consistent by extracting parseEthTopics
• cbor decode event log topic values
• Changed EthBlock.Extradata type to EthBytes

Additional Info

Checklist

Before you mark the PR ready for review, please make sure that:

• Commits have a clear commit message.
• PR title is in the form of of <PR type>: <area>: <change being made>
  • example: fix: mempool: Introduce a cache for valid signatures
  • PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
  • area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
• New features have usage guidelines and / or documentation updates in
  • Lotus Documentation
  • Discussion Tutorials
• Tests exist for new functionality or change in behavior
• CI is green

[magik6k]

Can we somehow avoid pulling all those deps in? According to scripts/gomod-diff.sh this is like ~54k lines of diff in dependencies

[iand]

Can we somehow avoid pulling all those deps in? According to scripts/gomod-diff.sh this is like ~54k lines of diff in dependencies

Yeah I have changed to a smaller json-schema validator which reduces new deps

[iand]

eth_getBlockByNumber validation failures:

    eth_conformance_test.go:306: - transactions: Must validate at least one schema (anyOf)
    eth_conformance_test.go:306: - transactions.0: Must validate one and only one schema (oneOf)
    eth_conformance_test.go:306: - transactions.0: accessList is required
    eth_conformance_test.go:306: - transactions.0: yParity is required
    eth_conformance_test.go:306: - extraData: Does not match pattern '^0x[0-9a-f]*$'  (fixed)
    eth_conformance_test.go:309: response did not validate

[iand]

eth_getBlockByHash with full transaction validation failures:

    eth_conformance_test.go:306: - extraData: Does not match pattern '^0x[0-9a-f]*$'
    eth_conformance_test.go:306: - transactions: Must validate at least one schema (anyOf)
    eth_conformance_test.go:306: - transactions.0: Must validate one and only one schema (oneOf)
    eth_conformance_test.go:306: - transactions.0: accessList is required
    eth_conformance_test.go:306: - transactions.0: yParity is required
    eth_conformance_test.go:309: response did not validate

[iand]

eth_getTransactionByBlockHashAndIndex and eth_getTransactionByBlockNumberAndIndex validation failures.

Note that we don't implement these methods but simply return the zero value of ethtypes.EthTx

    eth_conformance_test.go:324: - (root): Must validate one and only one schema (oneOf)
    eth_conformance_test.go:324: - (root): accessList is required
    eth_conformance_test.go:324: - (root): yParity is required
    eth_conformance_test.go:324: - to: Invalid type. Expected: string, given: null
    eth_conformance_test.go:324: - blockHash: Invalid type. Expected: string, given: null
    eth_conformance_test.go:324: - blockNumber: Invalid type. Expected: string, given: null
    eth_conformance_test.go:324: - transactionIndex: Invalid type. Expected: string, given: null
    eth_conformance_test.go:327: response did not validate

[iand]

eth_getTransactionByHash similar to the failures from eth_getBlockByNumber

    eth_conformance_test.go:338: - (root): Must validate one and only one schema (oneOf)
    eth_conformance_test.go:338: - (root): accessList is required
    eth_conformance_test.go:338: - (root): yParity is required
    eth_conformance_test.go:341: response did not validate

[iand]

eth_getStorageAt gives a vm failure, but I may be calling it incorrectly.

I call EthGetStorageAt( address of contract, ethtypes.EthBytes{0}, "0x0")

Error:      	Received unexpected error:

Call failed: implicit message failed with exit code: 21 and error: message failed with backtrace:
00: f01003 (method 5) -- Serialization error for Cbor protocol: Msg("invalid length 66, expected at most 32 bytes") (21)

[iand]

eth_call gives a cbor decode failure from decoding invokeResult.MsgRct.Return in EthCall

Error:      	Received unexpected error:
            	expected cbor type 'byte string' in input

[iand]

With regards to the missing fields on Transactions the openrpc spec gives 3 possible response types for a call such as eth_getTransactionByHash:

1. Signed 1559 Transaction: missing accessList, yParity
2. Signed 2930 Transaction: missing accessList, gasPrice, yParity
3. Signed Legacy Transaction: missing gasPrice

See https://www.jsonschemavalidator.net/s/euWIlkpO

I'm not sure which transaction type we are aiming to support

[mur-me]

Hi Lotus devs!

Users have found regression for params field in the input json after update to the v1.20.0-hyperspace-0131:

• params became mandatory, but shouldn't - eth-go is broken by eth_chainId
• params couldn't be null - broken eth_maxPriorityFeePerGas

Previously params can be in the following conditions:

• params is optional field - request could be without it
• params: [] - empty
• params: null
• params: [ array with something meaningful inside - array, string, object, etc]

Please check this issue

[iand]

Flaky test TestEthOpenRPCConformance is covered by #10170

[magik6k]

Looks good, will merge after tests pass (the fevm itest is broken on the release branch so not blocking)

[Stebalien]

That schema is a lie. See NethermindEth/nethermind#3276