Merge F2F #2620

paulhauner · 2021-09-23T23:23:39Z

Overview

This is the canonical "Merge F2F" branch for interop testing.

Credits to @ethDreamer for starting this branch.

Notes to Interop Teams

We're keeping scripts that can run Lighthouse testnets against the EL clients at sig/lighthouse-merge-f2f.

Building Lighthouse from Source

You'll need to install Rust, we recommend using the standard rustup tool. The commands here will install the binaries to ~/.cargo/bin, so you'll need to make sure that's on your PATH.

Once you have installed Rust, install Lighthouse with:

git clone [email protected]:sigp/lighthouse.git
cd lighthouse
git checkout merge-f2f
make

Optional: to get the lcli binary, run make install-lcli.

Lighthouse via Docker

Our CI will update the sigp/lighthouse:merge-f2f branch whenever this PR is updated. Run it with:

docker run sigp/lighthouse:merge-f2f lighthouse --version

Sometimes we use lcli (Lighthouse CLI) for creating testnets. You can also find it on Dockerhub at sigp/lcli:merge-f2f.

Resources

Consensus Specifications: https://github.com/ethereum/consensus-specs/tree/v1.1.0/specs/merge
Engine API Subset for the Merge Interop: https://github.com/ethereum/execution-apis/blob/v1.0.0-alpha.2/src/engine/interop/specification.md
Scripts for starting LH testnets: https://github.com/sigp/lighthouse-merge-f2f

Ground Rules

This section contains some ground-rules for developing and using this PR.

Avoid pushing changes directly, prefer to create a feature PR on sigp/lighthouse (this repo) with merge-f2f (this branch) as the target branch.
When merging changes, do a squash merge via the Github UI (or something that creates the same commit messages).
PRs being merged here have a lower requirement for quality and review when compared to unstable. This is not production code, we have no user-facing guarantees for stability.
Expect this branch to get rebased onto unstable from time to time.

We should also try to extract functionality and merge it to unstable from time-to-time to avoid a big nasty merge into unstable.

TODO

Before Interop

All complete

During Interop

Fix failing arbitrary CI check (see comment).
End blocking tasks created by the ExecutionLayer when the shutdown signal is called.
Follow up once Clarification on get_pow_block Clarification on get_pow_block ethereum/consensus-specs#2636 is resolved.
Call preparePayload when block producer is known, prior to producing a block
Add per-endpoint caches for payload ids
Add tests with a terminal block hash override.
Add tests for newly added CLI flags.

Additional Info

This PR contains a new Github Action to push to Dockerhub whenever this branch is updated. We should remove that before merging into unstable.

Do not merge this branch!

paulhauner · 2021-09-27T00:44:48Z

I've just rebased this branch on the latest unstable, primarily in response to #2579 being merged into unstable.

It was a rather hairy rebase, since the commits from #2579 were applied to this branch in non-consecutive commits. I ended up checking out unstable, then manually cherry-picking each non-#2579 commit onto this branch. Then, I squashed all those commits into 42f04b0 and then cherry-picked the remaining 3 squash-merge PRs on top of that (#2622, #2626, #2630).

Ultimately, I think we should end up with the same set of changes we had before. In case anything was lost, I've made a backup of this branch before my big rebase here: https://github.com/paulhauner/lighthouse/tree/merge-f2f-backup

paulhauner · 2021-09-28T22:55:29Z

I just force-pushed to rebase onto unstable to fix a conflict introduced by #2632 and also to remove an mistake merge (not squash merge) commit.

paulhauner · 2021-10-01T05:53:01Z

I just rebased onto unstable to fix conflicts in:

Cargo.lock
beacon_node/beacon_chain/src/beacon_chain.rs

paulhauner · 2021-10-11T23:15:29Z

I have just merged a breaking change into this branch (#2676).

Previously, Lighthouse was very awkwardly parsing --terminal-total-difficulty-override as non-0x-prefixed-hex. So, if you provided --terminal-total-difficulty-override 16 , you were actually providing 22 decimal.

With #2676, we now accept only decimal values (no hex, regardless of 0x-prefix). So, from now on --terminal-total-difficulty-override 16 mean 16 decimal (not 22 decimal).

I can't do this in a backwards-compatible way, so any setup using --terminal-total-difficulty-override is probably broken. I'm very sorry for this 🙏. We would need to "rip the band-aid off" at some point, and sooner is better than later IMO.

paulhauner · 2021-10-12T00:31:36Z

I've just rebased on unstable and force-pushed. There were no notable conflicts.

paulhauner · 2021-10-12T04:15:47Z

In order to get Arbitrary tests working it looks like we'll need to do the equivalent of this PR, but for ethereum_types::Uint256 (and similar types):

paritytech/parity-common#378

MariusVanDerWijden · 2021-10-15T08:51:13Z

I get an ForkChoiceError(InvalidProtoArrayBytes("Failed to decode ProtoArrayForkChoice: OffsetOutOfBounds(60562903)")) after updating to the newest version

Edit: ah nvm nuking the db and restarting worked

paulhauner · 2021-10-16T03:59:12Z

I get an ForkChoiceError(InvalidProtoArrayBytes("Failed to decode ProtoArrayForkChoice: OffsetOutOfBounds(60562903)")) after updating to the newest version

This was a breaking DB change documented #2686.

Apologies for the inconvenience. Nuking the DB is indeed the correct solution. It won't happen again after that.

Added Execution Payload from Rayonism Fork Updated new Containers to match Merge Spec Updated BeaconBlockBody for Merge Spec Completed updating BeaconState and BeaconBlockBody Modified ExecutionPayload<T> to use Transaction<T> Mostly Finished Changes for beacon-chain.md Added some things for fork-choice.md Update to match new fork-choice.md/fork.md changes ran cargo fmt Added Missing Pieces in eth2_libp2p for Merge fix ef test Various Changes to Conform Closer to Merge Spec

* Update to v1.1.0-beta.4 (squash of #2548) * SSZ, cached tree hash, EF tests

* Remove unchecked arith from ssz_derive * Address clippy lints in block_verfication * Use safe math for is_valid_gas_limit

* - Update the fork choice `ProtoNode` to include `is_merge_complete` - Add database migration for the persisted fork choice * update tests * Small cleanup * lints * store execution block hash in fork choice rather than bool

* Fixed Gossip Topics on Fork Boundary

* Gossip Block Validation is Much More Efficient Co-authored-by: realbigsean <[email protected]>

* Checkout serde_utils from rayonism * Make eth1::http functions pub * Add bones of execution_layer * Modify decoding * Expose Transaction, cargo fmt * Add executePayload * Add all minimal spec endpoints * Start adding json rpc wrapper * Finish custom JSON response handler * Switch to new rpc sending method * Add first test * Fix camelCase * Finish adding tests * Begin threading execution layer into BeaconChain * Fix clippy lints * Fix clippy lints * Thread execution layer into ClientBuilder * Add CLI flags * Add block processing methods to ExecutionLayer * Add block_on to execution_layer * Integrate execute_payload * Add extra_data field * Begin implementing payload handle * Send consensus valid/invalid messages * Fix minor type in task_executor * Call forkchoiceUpdated * Add search for TTD block * Thread TTD into execution layer * Allow producing block with execution payload * Add LRU cache for execution blocks * Remove duplicate 0x on ssz_types serialization * Add tests for block getter methods * Add basic block generator impl * Add is_valid_terminal_block to EL * Verify merge block in block_verification * Partially implement --terminal-block-hash-override * Add terminal_block_hash to ChainSpec * Remove Option from terminal_block_hash in EL * Revert merge changes to consensus/fork_choice * Remove commented-out code * Add bones for handling RPC methods on test server * Add first ExecutionLayer tests * Add testing for finding terminal block * Prevent infinite loops * Add insert_merge_block to block gen * Add block gen test for pos blocks * Start adding payloads to block gen * Fix clippy lints * Add execution payload to block gen * Add execute_payload to block_gen * Refactor block gen * Add all routes to mock server * Use Uint256 for base_fee_per_gas * Add working execution chain build * Remove unused var * Revert "Use Uint256 for base_fee_per_gas" This reverts commit 6c88f19. * Fix base_fee_for_gas Uint256 * Update execute payload handle * Improve testing, fix bugs * Fix default fee-recipient * Fix fee-recipient address (again) * Add check for terminal block, add comments, tidy * Apply suggestions from code review Co-authored-by: realbigsean <[email protected]> * Fix is_none on handle Drop * Remove commented-out tests Co-authored-by: realbigsean <[email protected]>

* Start implemented serde for transactions * Revise serde impl * Add tests for transaction decoding

* Add geth request vectors * Add geth response vectors * Fix clippy lints

* Start adding merge tests * Expose MockExecutionLayer * Add mock_execution_layer to BeaconChainHarness * Progress with merge test * Return more detailed errors with gas limit issues * Use a better gas limit in block gen * Ensure TTD is met in block gen * Fix basic_merge tests * Start geth testing * Fix conflicts after rebase * Remove geth tests * Improve merge test * Address clippy lints * Make pow block gen a pure function * Add working new test, breaking existing test * Fix test names * Add should_panic * Don't run merge tests in debug * Detect a tokio runtime when starting MockServer * Fix clippy lint, include merge tests

* Thread eth1_block_hash into interop genesis state * Add merge-fork-epoch flag * Build LH with minimal spec by default * Add verbose logs to execution_layer * Add --http-allow-sync-stalled flag * Update lcli new-testnet to create genesis state * Fix http test * Fix compile errors in tests

* Fixed bugs for m3 readiness * woops * cargo fmt..

* Reject some HTTP endpoints when EL is not ready * Restrict more endpoints * Add watchdog task * Change scheduling * Update to new schedule * Add "syncing" concept * Remove RequireSynced * Add is_merge_complete to head_info * Cache latest_head in Engines * Call consensus_forkchoiceUpdate on startup

* Ignore payload errors * Only return payload handle on valid response * Push some engine logs down to debug * Push ee fork choice log to debug * Push engine call failure to debug * Push some more errors to debug * Fix panic at startup

* Add payload verification status to fork choice * Pass payload verification status to import_block * Add valid back-propagation * Add head safety status latch to API * Remove ExecutionLayerStatus * Add execution info to client notifier * Update notifier logs * Change use of "hash" to refer to beacon block * Shutdown on invalid finalized block * Tidy, add comments * Fix failing FC tests * Allow blocks with unsafe head * Fix forkchoiceUpdate call on startup

* update initializing from eth1 for merge genesis * read execution payload header from file lcli * add `create-payload-header` command to `lcli` * fix base fee parsing * Apply suggestions from code review * default `execution_payload_header` bool to false when deserializing `meta.yml` in EF tests Co-authored-by: Paul Hauner <[email protected]>

paulhauner · 2021-10-27T01:06:29Z

I have just rebased this branch onto unstable. There should be no substantive changes.

paulhauner · 2021-11-03T05:01:00Z

I am closing this branch, but I will leave the merge-f2f branch on this repository for the foreseeable future.

This branch will be frozen with the purpose of providing support for the Pithos testnet.

We will not continue to develop atop this branch. Instead, we will be working on #2768 which targets a new series of specifications.

Thanks to everyone who contributed! Your work will live on in #2768 🚀

paulhauner added do-not-merge merge-f2f Relates to the Oct 2021 Merge F2F labels Sep 23, 2021

paulhauner force-pushed the merge-f2f branch from cce6f63 to 78fd388 Compare September 27, 2021 00:38

paulhauner force-pushed the merge-f2f branch from a11c92c to e559bd9 Compare September 28, 2021 22:53

paulhauner force-pushed the merge-f2f branch from aafcbcb to 306268d Compare October 1, 2021 05:52

paulhauner force-pushed the merge-f2f branch from 10df925 to 09a5d62 Compare October 12, 2021 00:31

ethDreamer and others added 16 commits October 27, 2021 12:03

Fix consensus, SSZ, tree hash & run merge EF tests (#2622)

82a1df5

* Update to v1.1.0-beta.4 (squash of #2548) * SSZ, cached tree hash, EF tests

Fix clippy lints on merge-f2f (#2626)

b7f094a

* Remove unchecked arith from ssz_derive * Address clippy lints in block_verfication * Use safe math for is_valid_gas_limit

Update merge consensus to v1.1.0-beta.5 (#2630)

817e1d8

Handle merge fork in web3signer (#2631)

820b92e

Update test vectors to v1.1.0 (#2642)

2f2267b

Store execution block hash in fork choice (#2643)

2415ccc

* - Update the fork choice `ProtoNode` to include `is_merge_complete` - Add database migration for the persisted fork choice * update tests * Small cleanup * lints * store execution block hash in fork choice rather than bool

Fork boundary fix (#2646)

bbba443

* Fixed Gossip Topics on Fork Boundary

Finished Gossip Block Validation Conditions (#2640)

3801673

* Gossip Block Validation is Much More Efficient Co-authored-by: realbigsean <[email protected]>

[Merge] Add serde impls for Transactions type (#2649)

c6d89a2

* Start implemented serde for transactions * Revise serde impl * Add tests for transaction decoding

[Merge] Add execution API test vectors from Geth (#2651)

44db773

* Add geth request vectors * Add geth response vectors * Fix clippy lints

add automated docker build for merge-f2f branch (#2654)

445d71c

Add merge fork_epoch and fork_version to Config (#2663)

8237853

Disable autotests for beacon_chain (#2658)

d2f5757

paulhauner and others added 10 commits October 27, 2021 12:04

Fixed bugs for m3 readiness (#2669)

5421ab5

* Fixed bugs for m3 readiness * woops * cargo fmt..

Update lcli pubkey replace command (#2677)

1fdb21b

Disable notifier logging from dummy eth1 backend (#2680)

45e2c32

[Merge] Optimistic EL verification (#2683)

f8e9154

* Ignore payload errors * Only return payload handle on valid response * Push some engine logs down to debug * Push ee fork choice log to debug * Push engine call failure to debug * Push some more errors to debug * Fix panic at startup

Fix db paths when datadir is relative (#2682)

95747ce

Accept TTD override as decimal (#2676)

06bcbdd

paulhauner force-pushed the merge-f2f branch from 09a5d62 to 06bcbdd Compare October 27, 2021 01:06

paulhauner mentioned this pull request Nov 3, 2021

Kintsugi 🍵 #2768

Merged

paulhauner closed this Nov 3, 2021

paulhauner deleted the merge-f2f branch November 24, 2022 02:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge F2F #2620

Merge F2F #2620

paulhauner commented Sep 23, 2021 •

edited by ethDreamer

Loading

paulhauner commented Sep 27, 2021

paulhauner commented Sep 28, 2021

paulhauner commented Oct 1, 2021

paulhauner commented Oct 11, 2021

paulhauner commented Oct 12, 2021

paulhauner commented Oct 12, 2021 •

edited

Loading

MariusVanDerWijden commented Oct 15, 2021 •

edited

Loading

paulhauner commented Oct 16, 2021

paulhauner commented Oct 27, 2021

paulhauner commented Nov 3, 2021

Merge F2F #2620

Merge F2F #2620

Conversation

paulhauner commented Sep 23, 2021 • edited by ethDreamer Loading

Overview

Notes to Interop Teams

Building Lighthouse from Source

Lighthouse via Docker

Resources

Ground Rules

TODO

Before Interop

During Interop

Additional Info

paulhauner commented Sep 27, 2021

paulhauner commented Sep 28, 2021

paulhauner commented Oct 1, 2021

paulhauner commented Oct 11, 2021

paulhauner commented Oct 12, 2021

paulhauner commented Oct 12, 2021 • edited Loading

MariusVanDerWijden commented Oct 15, 2021 • edited Loading

paulhauner commented Oct 16, 2021

paulhauner commented Oct 27, 2021

paulhauner commented Nov 3, 2021

paulhauner commented Sep 23, 2021 •

edited by ethDreamer

Loading

paulhauner commented Oct 12, 2021 •

edited

Loading

MariusVanDerWijden commented Oct 15, 2021 •

edited

Loading