docs: clean up simulation-test and move integration stuff out to plac…

…eholder file
ChainSafe · Dec 10, 2023 · 8806bd9 · 8806bd9
1 parent a49c439
commit 8806bd9
Show file tree

Hide file tree

Showing 2 changed files with 121 additions and 49 deletions.
diff --git a/docs/pages/contribution/testing/integration-tests.md b/docs/pages/contribution/testing/integration-tests.md
@@ -0,0 +1,27 @@
+# Integration Tests
+
+The following tests are found in `packages/beacon-node`
+
+#### `test:sim:withdrawals`
+
+This test simulates capella blocks with withdrawals. It tests lodestar against Geth and EthereumJS.
+
+There are two ENV variables that are required to run this test:
+
+- `EL_BINARY_DIR`: the docker image setup to handle the test case
+- `EL_SCRIPT_DIR`: the script that will be used to start the EL client. All of the scripts can be found in `packages/beacon-node/test/scripts/el-interop` and the `EL_SCRIPT_DIR` is the sub-directory name in that root that should be used to run the test.
+
+The command to run this test is:
+
+`EL_BINARY_DIR=g11tech/geth:withdrawals EL_SCRIPT_DIR=gethdocker yarn mocha test/sim/withdrawal-interop.test.ts`
+
+The images used by this test during CI are:
+
+- `GETH_WITHDRAWALS_IMAGE: g11tech/geth:withdrawalsfeb8`
+- `ETHEREUMJS_WITHDRAWALS_IMAGE: g11tech/ethereumjs:blobs-b6b63`
+
+#### `test:sim:merge-interop`
+
+#### `test:sim:mergemock`
+
+#### `yarn test:sim:blobs`
diff --git a/docs/pages/contribution/testing/simulation-tests.md b/docs/pages/contribution/testing/simulation-tests.md
@@ -6,6 +6,52 @@ At a very high level, simulation testing will setup a testnet from genesis and l
 
 These tests usually setup full testnets with multiple consensus clients and their paired execution node. In many instance we are looking to just exercise the Lodestar code but there are some places where there is also testing to see how Lodestar works in relation to the other consensus clients, like Lighthouse. As you can imagine, there is quite a bit of machinery that is responsible for setting up and managing the simulations and assertions. This section will help to go over those bits and pieces. Many, but not all, of these classes can be found in `packages/cli/test/utils/simulation`.
 
+## Running Sim Tests
+
+There are a number of sim tests that are available and each has a slightly different purpose. All are run by CI and must pass for a PR to be valid for merging. Most tests require a couple of environment variables to be set.
+
+### Environment Variables
+
+To see what typical values for these are check out the `test-sim.yaml` workflow file in the `.github/workflows` directory.
+
+- `GETH_DOCKER_IMAGE`: The geth docker image that will be used
+- `NETHERMIND_IMAGE`: The nethermind docker image that will be used
+- `LIGHTHOUSE_IMAGE`: The lighthouse docker image that will be used
+
+### `test:sim:multifork`
+
+The multi-fork sim test checks most of the functionality Lodestar provides. Is verifies that Lodestar is capable of peering, moving through all of the forks and using various sync methods in a testnet environment. Lodestar is tested with both Geth and Nethermind as the execution client. It also checks a Lighthouse/Geth node for cross client compatibility.
+
+```sh
+GETH_DOCKER_IMAGE=ethereum/client-go:v1.11.6 \
+    LIGHTHOUSE_DOCKER_IMAGE=sigp/lighthouse:latest-amd64-modern-dev \
+    NETHERMIND_DOCKER_IMAGE=nethermind/nethermind:1.18.0 \
+    yarn workspace @chainsafe/lodestar test:sim:multifork
+```
+
+### `test:sim:endpoints`
+
+This tests that various endpoints of the beacon node and validator client are working as expected.
+
+```sh
+GETH_DOCKER_IMAGE=ethereum/client-go:v1.11.6 \
+    yarn workspace @chainsafe/lodestar test:sim:endpoints
+```
+
+### `test:sim:deneb`
+
+This test is still included in our CI but is no longer as important as it once was. Lodestar is often the first client to implement new features and this test was created before geth was upgraded with the features required to support the Deneb fork. To test that Lodestar was ready this test uses mocked geth instances. It is left as a placeholder for when the next fork comes along that requires a similar approach.
+
+### `test:sim:mixedcleint`
+
+Checks that Lodestar is compatible with other consensus validators and vice-versa. All tests use Geth as the EL.
+
+```sh
+GETH_DOCKER_IMAGE=ethereum/client-go:v1.11.6 \
+    LIGHTHOUSE_DOCKER_IMAGE=sigp/lighthouse:latest-amd64-modern-dev \
+    yarn workspace @chainsafe/lodestar test:sim:mixedclient
+```
+
 ## Sim Test Infrastructure
 
 When setting up and running the simulations, interactions with the nodes is through the published node API's. All functionality is actuated via http request and by "plugging in" this way it is possible to run the nodes in a stand-alone fashion, as they would be run in production, but to still achieve a tightly monitored and controlled environment. If code needs to be executed on a "class by class" basis or with mocking involved then the test is not a simulation test and would fall into one of the other testing categories. See the [Testing](../testing.md) page for more information on the other types of tests available for Lodestar.
@@ -37,60 +83,59 @@ There are a number of assertions that are added to simulations by default. They
 
 Because of the flexibility, and complexity, there is a section specifically for how to create custom assertions below. See [custom assertions](#custom-assertions) for more info.
 
-### Simulation Reports
-
-### Simulation Logging
-
-## Running Sim Tests
-
-There are a number of sim tests that are available and each has a slightly different purpose. All are run by CI and must pass for a PR to be valid for merging.
-
-### `test:sim:multifork`
-
-The multifork sim test checks that Lodestar is capable of moving through all forks in a testnet. Lodestar is tested with Geth and Nethermind as the execution client. It also checks a Lighthouse/Geth node for cross client compatibility. The test is run in several phases where each epoch is responsible for different actions. The phases of the test are as follows:
-
-- Epoch 1
-
-`GETH_DOCKER_IMAGE=ethereum/client-go:v1.11.6 LIGHTHOUSE_DOCKER_IMAGE=sigp/lighthouse:latest-amd64-modern-dev NETHERMIND_DOCKER_IMAGE=nethermind/nethermind:1.18.0 yarn test:sim:multifork`
-
-### `packages/cli`
+### Custom Assertions
 
-- `yarn test:sim:mixedclient`
-- `yarn test:sim:endpoints`
-- `yarn test:sim:deneb`
-- `yarn test:sim:backup_eth_provider`
+Check back soon for more information on how to create custom assertions.
 
-## Custom Assertions
-
-The following tests are found in `packages/beacon-node`
-
-#### `test:sim:withdrawals`
-
-This test simulates capella blocks with withdrawals. It tests lodestar against Geth and EthereumJS.
-
-There are two ENV variables that are required to run this test:
-
-- `EL_BINARY_DIR`: the docker image setup to handle the test case
-- `EL_SCRIPT_DIR`: the script that will be used to start the EL client. All of the scripts can be found in `packages/beacon-node/test/scripts/el-interop` and the `EL_SCRIPT_DIR` is the sub-directory name in that root that should be used to run the test.
-
-The command to run this test is:
-
-`EL_BINARY_DIR=g11tech/geth:withdrawals EL_SCRIPT_DIR=gethdocker yarn mocha test/sim/withdrawal-interop.test.ts`
-
-The images used by this test during CI are:
+### Simulation Reports
 
-- `GETH_WITHDRAWALS_IMAGE: g11tech/geth:withdrawalsfeb8`
-- `ETHEREUMJS_WITHDRAWALS_IMAGE: g11tech/ethereumjs:blobs-b6b63`
+Sim tests that are run using the simulation framework output a table of information to the console. The table summarizes the state of all of the nodes and the network at each slot.
+
+Here is an example of the table and how to interpret it:
+
+```sh
+┼─────────────────────────────────────────────────────────────────────────────────────────────────┼
+│ fork       │ eph   │ slot │ head       │ finzed   │ peers    │ attCount │ incDelay │ errors     │
+┼─────────────────────────────────────────────────────────────────────────────────────────────────┼
+│ capella    │ 9/0   │ 72   │ 0x95c4..   │ 56       │ 3        │ 16       │ 1.00     │ 0          │
+│ capella    │ 9/1   │ 73   │ 0x9dfc..   │ 56       │ 3        │ 16       │ 1.00     │ 0          │
+│ capella    │ 9/2   │ 74   │ 0xdf3f..   │ 56       │ 3        │ 16       │ 1.00     │ 0          │
+│ capella    │ 9/3   │ 75   │ 0xbeae..   │ 56       │ 3        │ 16       │ 1.00     │ 0          │
+│ capella    │ 9/4   │ 76   │ 0x15fa..   │ 56       │ 3        │ 16       │ 1.00     │ 0          │
+│ capella    │ 9/5   │ 77   │ 0xf8ff..   │ 56       │ 2,3,3,2  │ 16       │ 1.00     │ 0          │
+│ capella    │ 9/6   │ 78   │ 0x8199..   │ 56       │ 2,3,3,2  │ 16       │ 1.20     │ 0          │
+│ capella    │ 9/7   │ 79   │ different  │ 56       │ 2,3,3,2  │ 16       │ 1.50     │ 2          │
+┼─────────────────────────────────────────────────────────────────────────────────────────────────┼
+│ Att Participation: H: 0.75, S: 1.00, T: 0.75 - SC Participation: 1.00                           │
+┼─────────────────────────────────────────────────────────────────────────────────────────────────┼
+```
+
+#### Slot Information
+
+- `fork`: shows what fork is currently being tested
+- `eph`: During simulation tests the Lodestar repo is setup to use 8 slot per epoch so what is shown is the epoch number and the slot number within that epoch as `epoch/slot`
+- `slot`: The slot number that is currently being processed
+- `head`: If all clients have the the same head the first couple of bytes of the hash are shown. If all clients do not have the same head `different` is reported.
+- `finzed`: Shows the number of the last finalized slot
+- `peers`: The number of peers that each node is connected to. If all have the same number then only a single value is shown. If they do not have the same number of peers count for each node is reported in a comma-separated list
+- `attCount`: The number of attestations that the node has seen.
+- `incDelay`: The average number of slots inclusion delay was experienced for the attestations. Often attestations for the current head arrive more than one slot behind and this value tracks that
+- `errors`: The number of errors that were encountered during the slot
+
+#### Epoch Information
+
+- `H`: The percentage of nodes, at epoch transition, that voted for the head block
+- `S`: The percentage of nodes, at epoch transition, that voted for the source block
+- `T`: The percentage of nodes, at epoch transition, that voted for the target block
+- `SC Participation`: The sync committee participation rate
 
-#### `test:sim:merge-interop`
+### Simulation Logging
 
-#### `test:sim:mergemock`
+The simulation environment will capture all of the logs from all nodes that are running. The logs can be found in the `packages/cli/test-logs` directory. The logs are named with the following convention:
 
-#### `yarn test:sim:blobs`
+`<PURPOSE>-<TYPE>_<CLIENT>.log`
 
-## Docker Images Used in Sim Testing
+Some examples are:
 
-- `GETH_IMAGE: ethereum/client-go:v1.10.25`
-- `NETHERMIND_IMAGE: nethermind/nethermind:1.14.3`
-- `MERGEMOCK_IMAGE: g11tech/mergemock:latest`
-- `ETHEREUMJS_BLOBS_IMAGE: g11tech/ethereumjs:blobs-b6b63`
+- `node-1-beacon_lodestar.log`: The is the first node in the simulation. It is the consensus layer. It is running the lodestar validator client.
+- `range-sync-execution_geth.log`: This is the node that was added to test pulling history in range sync mode. It was the execution layer and was running the geth execution client.