Merge test coverage

[terencechain]

Merge test cases

Initial Syncing (beacon_blocks_by_range)

• BN(fresh) <-> EE(fresh) can sync to head
• BN(fresh) <-> EE(synced) can sync to head
• BN(close to synced) <->(fresh) can sync to head
• Validate when EE goes offline, the UX is still smooth
• Validate when EE times out, the UX is still smooth
• Validate BN can retrospectively validate terminal block for optimistic syncing
• Can optimistic sync through very long non-finality durations

Regular syncing (gossip-sub)

• Can submit a voluntary exit
• Can slash someone and be slashed
• Validate when EE goes offline, the UX is still smooth
• Validate when EE times out, the UX is still smooth

Emergency conditions

• Gracefully shut down when an invalid block is justified/finalized
• --safe-slots-to-import-optimistically works as expected
• TERMINAL_BLOCK_HASH works as expected

Validator

• Duties are prevented under an optimistic mode, ensure smooth UX
• Web3signer works post-merge
• Fee recipient works with BN default
• Fee recipient works with VC default from the config file
• Fee recipient works with VC from local config file
• Fee recipient works with VC from remote URL config file
• Fee recipient works with dynamically updating VC fee recipients
• Checkpoint sync works past-merge, or when EE is syncing and beacon is optimistic

APIs

• Test all the prysm API endpoints post-Bellatrix
• Test all the official beacon API endpoints post-Bellatrix

Stress

• Test large blocks
• Test shorter slot times

Performance

• Validate histogram of response times for the different API methods
• Validate pprof during various run-time conditions
• Validate Jaeger in Prysm post-merge when it comes to common, processing operations
• Validate performance during optimistic sync?
• Validate disk-size growth for both EL and CL

[SeaMonkey82]

Some initial thoughts based on testing done so far:

Merge test cases

Initial Syncing (beacon_blocks_by_range)

* [ ]  BN(fresh) <-> EE(fresh) can sync to head

* [ ]  BN(fresh) <-> EE(synced) can sync to head

* [ ]  BN(close to synced) <->(fresh) can sync to head

These are true for all EL clients except ethereumjs at the moment since it doesn't currently support optimistic sync, so if the CL gets ahead, the only way to get it to sync is to start fresh. Resuming sync is something that gets tested usually more than once per day, as I'm always checking for the latest commits. If there are conditions which present potential changes in behavior specifically for a fresh sync, please let me know.

Regular syncing (gossip-sub)

* [ ]  Validate when EE goes offline, the UX is still smooth

This isn't currently true. Lots of error noise.

[2022-04-18 21:10:00] ERROR sync: Could not handle p2p pubsub error=Post "http://localhost:8578": dial tcp 127.0.0.1:8578: connect: connection refused
got an unexpected error
github.com/prysmaticlabs/prysm/beacon-chain/powchain.handleRPCError
        beacon-chain/powchain/engine_client.go:290
github.com/prysmaticlabs/prysm/beacon-chain/powchain.(*Service).NewPayload
        beacon-chain/powchain/engine_client.go:71
github.com/prysmaticlabs/prysm/beacon-chain/blockchain.(*Service).notifyNewPayload
        beacon-chain/blockchain/optimistic_sync.go:136
github.com/prysmaticlabs/prysm/beacon-chain/blockchain.(*Service).onBlock
        beacon-chain/blockchain/process_block.go:116
github.com/prysmaticlabs/prysm/beacon-chain/blockchain.(*Service).ReceiveBlock
        beacon-chain/blockchain/receive_block.go:39
github.com/prysmaticlabs/prysm/beacon-chain/sync.(*Service).beaconBlockSubscriber
        beacon-chain/sync/subscriber_beacon_blocks.go:34
github.com/prysmaticlabs/prysm/beacon-chain/sync.(*Service).subscribeWithBase.func1
        beacon-chain/sync/subscriber.go:190
runtime.goexit
        GOROOT/src/runtime/asm_amd64.s:1581
could not validate execution payload from execution engine
github.com/prysmaticlabs/prysm/beacon-chain/blockchain.(*Service).notifyNewPayload
        beacon-chain/blockchain/optimistic_sync.go:161
github.com/prysmaticlabs/prysm/beacon-chain/blockchain.(*Service).onBlock
        beacon-chain/blockchain/process_block.go:116
github.com/prysmaticlabs/prysm/beacon-chain/blockchain.(*Service).ReceiveBlock
        beacon-chain/blockchain/receive_block.go:39
github.com/prysmaticlabs/prysm/beacon-chain/sync.(*Service).beaconBlockSubscriber
        beacon-chain/sync/subscriber_beacon_blocks.go:34
github.com/prysmaticlabs/prysm/beacon-chain/sync.(*Service).subscribeWithBase.func1
        beacon-chain/sync/subscriber.go:190
runtime.goexit
        GOROOT/src/runtime/asm_amd64.s:1581
could not verify new payload
github.com/prysmaticlabs/prysm/beacon-chain/blockchain.(*Service).onBlock
        beacon-chain/blockchain/process_block.go:118
github.com/prysmaticlabs/prysm/beacon-chain/blockchain.(*Service).ReceiveBlock
        beacon-chain/blockchain/receive_block.go:39
github.com/prysmaticlabs/prysm/beacon-chain/sync.(*Service).beaconBlockSubscriber
        beacon-chain/sync/subscriber_beacon_blocks.go:34
github.com/prysmaticlabs/prysm/beacon-chain/sync.(*Service).subscribeWithBase.func1
        beacon-chain/sync/subscriber.go:190
runtime.goexit
        GOROOT/src/runtime/asm_amd64.s:1581
could not process block
github.com/prysmaticlabs/prysm/beacon-chain/blockchain.(*Service).ReceiveBlock
        beacon-chain/blockchain/receive_block.go:40
github.com/prysmaticlabs/prysm/beacon-chain/sync.(*Service).beaconBlockSubscriber
        beacon-chain/sync/subscriber_beacon_blocks.go:34
github.com/prysmaticlabs/prysm/beacon-chain/sync.(*Service).subscribeWithBase.func1
        beacon-chain/sync/subscriber.go:190
runtime.goexit
        GOROOT/src/runtime/asm_amd64.s:1581 topic=/eth2/e7acb210/beacon_block/ssz_snappy

Validator

* [ ]  Duties are prevented under an optimistic mode, ensure smooth UX

This works fine.

[SeaMonkey82]

Some notes on docs:

https://docs.prylabs.network/docs/wallet/nondeterministic
The "Import validator accounts" section of this page is the only one that doesn't include a table of available flags. --keys-dir is given in the example, but --wallet-dir is omitted.

https://docs.prylabs.network/docs/how-prysm-works/ethereum-public-api
Page describes versions of the API spec as "v1, v2.0.0 and dev"
Current version of the spec is v2.2.0 and prior version is v2.1.0.

[SeaMonkey82]

Submitted a voluntary exit without incident today.

      "index": "110393",
      "balance": "32008291207",
      "status": "exited_unslashed",
      "validator": {
        "pubkey": "0xb55c538917ac6f096677b794ab52cf4c82c1431771f97693fbff189d7b3ffbaf218174562c8b9aa168ad7ba33df04b7e",
        "withdrawal_credentials": "0x00fd2d67e5cb6eac9f1019400267ecf9aad4570bec7fd25a130c884922f9621f",
        "effective_balance": "32000000000",
        "slashed": false,
        "activation_eligibility_epoch": "10932",
        "activation_epoch": "10938",
        "exit_epoch": "11265",
        "withdrawable_epoch": "11521"