extend REST config with more spec constants #2962

etan-status · 2021-10-06T10:14:08Z

So far, the REST/RPC config response did not include all spec constants.
The spec for /eth/v1/config/spec defines that the response should
include constants for all hard forks known by the beacon node. This
patch extends the corresponding responses to include more constants.

etan-status · 2021-10-06T10:14:24Z

Depends on #2958

etan-status · 2021-10-06T10:16:17Z

RPC @ Prater:

% curl -d '{"jsonrpc":"2.0","method":"get_v1_config_spec","params":[],"id":1}' -H 'Content-Type: application/json' localhost:9190 -s | jq

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "CONFIG_NAME": "mainnet",
    "MAX_COMMITTEES_PER_SLOT": "64",
    "TARGET_COMMITTEE_SIZE": "128",
    "MAX_VALIDATORS_PER_COMMITTEE": "2048",
    "SHUFFLE_ROUND_COUNT": "90",
    "HYSTERESIS_QUOTIENT": "4",
    "HYSTERESIS_DOWNWARD_MULTIPLIER": "1",
    "HYSTERESIS_UPWARD_MULTIPLIER": "5",
    "SAFE_SLOTS_TO_UPDATE_JUSTIFIED": "8",
    "MIN_DEPOSIT_AMOUNT": "1000000000",
    "MAX_EFFECTIVE_BALANCE": "32000000000",
    "EFFECTIVE_BALANCE_INCREMENT": "1000000000",
    "MIN_ATTESTATION_INCLUSION_DELAY": "1",
    "SLOTS_PER_EPOCH": "32",
    "MIN_SEED_LOOKAHEAD": "1",
    "MAX_SEED_LOOKAHEAD": "4",
    "EPOCHS_PER_ETH1_VOTING_PERIOD": "64",
    "SLOTS_PER_HISTORICAL_ROOT": "8192",
    "MIN_EPOCHS_TO_INACTIVITY_PENALTY": "4",
    "EPOCHS_PER_HISTORICAL_VECTOR": "65536",
    "EPOCHS_PER_SLASHINGS_VECTOR": "8192",
    "HISTORICAL_ROOTS_LIMIT": "16777216",
    "VALIDATOR_REGISTRY_LIMIT": "1099511627776",
    "BASE_REWARD_FACTOR": "64",
    "WHISTLEBLOWER_REWARD_QUOTIENT": "512",
    "PROPOSER_REWARD_QUOTIENT": "8",
    "INACTIVITY_PENALTY_QUOTIENT": "67108864",
    "MIN_SLASHING_PENALTY_QUOTIENT": "128",
    "PROPORTIONAL_SLASHING_MULTIPLIER": "1",
    "MAX_PROPOSER_SLASHINGS": "16",
    "MAX_ATTESTER_SLASHINGS": "2",
    "MAX_ATTESTATIONS": "128",
    "MAX_DEPOSITS": "16",
    "MAX_VOLUNTARY_EXITS": "16",
    "INACTIVITY_PENALTY_QUOTIENT_ALTAIR": "50331648",
    "MIN_SLASHING_PENALTY_QUOTIENT_ALTAIR": "64",
    "PROPORTIONAL_SLASHING_MULTIPLIER_ALTAIR": "2",
    "SYNC_COMMITTEE_SIZE": "512",
    "EPOCHS_PER_SYNC_COMMITTEE_PERIOD": "256",
    "MIN_SYNC_COMMITTEE_PARTICIPANTS": "1",
    "PRESET_BASE": "mainnet",
    "TERMINAL_TOTAL_DIFFICULTY": "115792089237316195423570985008687907853269984665640564039457584007913129638912",
    "TERMINAL_BLOCK_HASH": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "MIN_GENESIS_ACTIVE_VALIDATOR_COUNT": "16384",
    "MIN_GENESIS_TIME": "1614588812",
    "GENESIS_FORK_VERSION": "0x00001020",
    "GENESIS_DELAY": "1919188",
    "ALTAIR_FORK_VERSION": "0x01001020",
    "ALTAIR_FORK_EPOCH": "36660",
    "MERGE_FORK_VERSION": "0x02001020",
    "MERGE_FORK_EPOCH": "18446744073709551615",
    "SHARDING_FORK_VERSION": "0x03001020",
    "SHARDING_FORK_EPOCH": "18446744073709551615",
    "SECONDS_PER_SLOT": "12",
    "SECONDS_PER_ETH1_BLOCK": "14",
    "MIN_VALIDATOR_WITHDRAWABILITY_DELAY": "256",
    "SHARD_COMMITTEE_PERIOD": "256",
    "ETH1_FOLLOW_DISTANCE": "2048",
    "INACTIVITY_SCORE_BIAS": "4",
    "INACTIVITY_SCORE_RECOVERY_RATE": "16",
    "EJECTION_BALANCE": "16000000000",
    "MIN_PER_EPOCH_CHURN_LIMIT": "4",
    "CHURN_LIMIT_QUOTIENT": "65536",
    "DEPOSIT_CHAIN_ID": "5",
    "DEPOSIT_NETWORK_ID": "5",
    "DEPOSIT_CONTRACT_ADDRESS": "0xff50ed3d0ec03ac01d4c79aad74928bff48a7b2b",
    "BLS_WITHDRAWAL_PREFIX": "0x00",
    "ETH1_ADDRESS_WITHDRAWAL_PREFIX": "0x01",
    "DOMAIN_BEACON_PROPOSER": "0x00000000",
    "DOMAIN_BEACON_ATTESTER": "0x01000000",
    "DOMAIN_RANDAO": "0x02000000",
    "DOMAIN_DEPOSIT": "0x03000000",
    "DOMAIN_VOLUNTARY_EXIT": "0x04000000",
    "DOMAIN_SELECTION_PROOF": "0x05000000",
    "DOMAIN_AGGREGATE_AND_PROOF": "0x06000000",
    "DOMAIN_SYNC_COMMITTEE": "0x07000000",
    "DOMAIN_SYNC_COMMITTEE_SELECTION_PROOF": "0x08000000",
    "DOMAIN_CONTRIBUTION_AND_PROOF": "0x09000000",
    "TARGET_AGGREGATORS_PER_COMMITTEE": "16",
    "RANDOM_SUBNETS_PER_VALIDATOR": "1",
    "EPOCHS_PER_RANDOM_SUBNET_SUBSCRIPTION": "256",
    "ATTESTATION_SUBNET_COUNT": "64",
    "TARGET_AGGREGATORS_PER_SYNC_SUBCOMMITTEE": "16",
    "SYNC_COMMITTEE_SUBNET_COUNT": "4"
  }
}

etan-status · 2021-10-06T10:17:01Z

REST @ Prater:

 % curl localhost:5052/eth/v1/config/spec -s | jq
{
  "data": {
    "CONFIG_NAME": "mainnet",
    "MAX_COMMITTEES_PER_SLOT": "64",
    "TARGET_COMMITTEE_SIZE": "128",
    "MAX_VALIDATORS_PER_COMMITTEE": "2048",
    "SHUFFLE_ROUND_COUNT": "90",
    "HYSTERESIS_QUOTIENT": "4",
    "HYSTERESIS_DOWNWARD_MULTIPLIER": "1",
    "HYSTERESIS_UPWARD_MULTIPLIER": "5",
    "SAFE_SLOTS_TO_UPDATE_JUSTIFIED": "8",
    "MIN_DEPOSIT_AMOUNT": "1000000000",
    "MAX_EFFECTIVE_BALANCE": "32000000000",
    "EFFECTIVE_BALANCE_INCREMENT": "1000000000",
    "MIN_ATTESTATION_INCLUSION_DELAY": "1",
    "SLOTS_PER_EPOCH": "32",
    "MIN_SEED_LOOKAHEAD": "1",
    "MAX_SEED_LOOKAHEAD": "4",
    "EPOCHS_PER_ETH1_VOTING_PERIOD": "64",
    "SLOTS_PER_HISTORICAL_ROOT": "8192",
    "MIN_EPOCHS_TO_INACTIVITY_PENALTY": "4",
    "EPOCHS_PER_HISTORICAL_VECTOR": "65536",
    "EPOCHS_PER_SLASHINGS_VECTOR": "8192",
    "HISTORICAL_ROOTS_LIMIT": "16777216",
    "VALIDATOR_REGISTRY_LIMIT": "1099511627776",
    "BASE_REWARD_FACTOR": "64",
    "WHISTLEBLOWER_REWARD_QUOTIENT": "512",
    "PROPOSER_REWARD_QUOTIENT": "8",
    "INACTIVITY_PENALTY_QUOTIENT": "67108864",
    "MIN_SLASHING_PENALTY_QUOTIENT": "128",
    "PROPORTIONAL_SLASHING_MULTIPLIER": "1",
    "MAX_PROPOSER_SLASHINGS": "16",
    "MAX_ATTESTER_SLASHINGS": "2",
    "MAX_ATTESTATIONS": "128",
    "MAX_DEPOSITS": "16",
    "MAX_VOLUNTARY_EXITS": "16",
    "INACTIVITY_PENALTY_QUOTIENT_ALTAIR": "50331648",
    "MIN_SLASHING_PENALTY_QUOTIENT_ALTAIR": "64",
    "PROPORTIONAL_SLASHING_MULTIPLIER_ALTAIR": "2",
    "SYNC_COMMITTEE_SIZE": "512",
    "EPOCHS_PER_SYNC_COMMITTEE_PERIOD": "256",
    "MIN_SYNC_COMMITTEE_PARTICIPANTS": "1",
    "PRESET_BASE": "mainnet",
    "TERMINAL_TOTAL_DIFFICULTY": "115792089237316195423570985008687907853269984665640564039457584007913129638912",
    "TERMINAL_BLOCK_HASH": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "MIN_GENESIS_ACTIVE_VALIDATOR_COUNT": "16384",
    "MIN_GENESIS_TIME": "1614588812",
    "GENESIS_FORK_VERSION": "0x00001020",
    "GENESIS_DELAY": "1919188",
    "ALTAIR_FORK_VERSION": "0x01001020",
    "ALTAIR_FORK_EPOCH": "36660",
    "MERGE_FORK_VERSION": "0x02001020",
    "MERGE_FORK_EPOCH": "18446744073709551615",
    "SHARDING_FORK_VERSION": "0x03001020",
    "SHARDING_FORK_EPOCH": "18446744073709551615",
    "SECONDS_PER_SLOT": "12",
    "SECONDS_PER_ETH1_BLOCK": "14",
    "MIN_VALIDATOR_WITHDRAWABILITY_DELAY": "256",
    "SHARD_COMMITTEE_PERIOD": "256",
    "ETH1_FOLLOW_DISTANCE": "2048",
    "INACTIVITY_SCORE_BIAS": "4",
    "INACTIVITY_SCORE_RECOVERY_RATE": "16",
    "EJECTION_BALANCE": "16000000000",
    "MIN_PER_EPOCH_CHURN_LIMIT": "4",
    "CHURN_LIMIT_QUOTIENT": "65536",
    "DEPOSIT_CHAIN_ID": "5",
    "DEPOSIT_NETWORK_ID": "5",
    "DEPOSIT_CONTRACT_ADDRESS": "0xff50ed3d0ec03ac01d4c79aad74928bff48a7b2b",
    "BLS_WITHDRAWAL_PREFIX": "0x00",
    "ETH1_ADDRESS_WITHDRAWAL_PREFIX": "0x01",
    "DOMAIN_BEACON_PROPOSER": "0x00000000",
    "DOMAIN_BEACON_ATTESTER": "0x01000000",
    "DOMAIN_RANDAO": "0x02000000",
    "DOMAIN_DEPOSIT": "0x03000000",
    "DOMAIN_VOLUNTARY_EXIT": "0x04000000",
    "DOMAIN_SELECTION_PROOF": "0x05000000",
    "DOMAIN_AGGREGATE_AND_PROOF": "0x06000000",
    "DOMAIN_SYNC_COMMITTEE": "0x07000000",
    "DOMAIN_SYNC_COMMITTEE_SELECTION_PROOF": "0x08000000",
    "DOMAIN_CONTRIBUTION_AND_PROOF": "0x09000000",
    "TARGET_AGGREGATORS_PER_COMMITTEE": "16",
    "RANDOM_SUBNETS_PER_VALIDATOR": "1",
    "EPOCHS_PER_RANDOM_SUBNET_SUBSCRIPTION": "256",
    "ATTESTATION_SUBNET_COUNT": "64",
    "TARGET_AGGREGATORS_PER_SYNC_SUBCOMMITTEE": "16",
    "SYNC_COMMITTEE_SUBNET_COUNT": "4"
  }
}

etan-status · 2021-10-06T10:19:47Z

beacon_chain/rpc/rest_config_api.nim

@@ -22,22 +22,16 @@ proc installConfigApiHandlers*(router: var RestRouter, node: BeaconNode) =
        (
          CONFIG_NAME:
            const_preset,
+
+          # https://github.com/ethereum/consensus-specs/blob/v1.1.1/presets/mainnet/phase0.yaml


Reordered these sections to be consistent with the order in the spec.

Makes it easier to avoid missing out adding new constants in the future, as the problem will be seen when updating the spec URLs over time. Also, makes it easier for developer manually looking into the JSON output to compare it against the spec as it is the same order.

beacon_chain/spec/eth2_apis/eth2_rest_serialization.nim

etan-status · 2021-10-06T10:25:18Z

beacon_chain/rpc/rest_config_api.nim

+          # GENESIS_SLOT
+          # GENESIS_EPOCH
+          # FAR_FUTURE_EPOCH
+          # BASE_REWARDS_PER_EPOCH
+          # DEPOSIT_CONTRACT_TREE_DEPTH
+          # JUSTIFICATION_BITS_LENGTH
+          # ENDIANNESS


Not sure about these. As they are phase0 ones and were not needed so far, I assume these were left out intentionally, so I did not add them.

Also left out on prysm

etan-status · 2021-10-06T10:25:46Z

beacon_chain/rpc/rest_config_api.nim

+          # TIMELY_SOURCE_FLAG_INDEX
+          # TIMELY_TARGET_FLAG_INDEX
+          # TIMELY_HEAD_FLAG_INDEX
+          # TIMELY_SOURCE_WEIGHT
+          # TIMELY_TARGET_WEIGHT
+          # TIMELY_HEAD_WEIGHT
+          # SYNC_REWARD_WEIGHT
+          # PROPOSER_WEIGHT
+          # WEIGHT_DENOMINATOR


Likewise, these are not part of the preset, so I only added the DOMAIN ones as they were added for phase0 in the past.

prysm is including these, with the exception of PARTICIPATION_FLAG_WEIGHTS further down. Will also include them for compatibility.

etan-status · 2021-10-06T10:26:07Z

beacon_chain/spec/presets.nim

  BLS_WITHDRAWAL_PREFIX*: byte = 0
+  ETH1_ADDRESS_WITHDRAWAL_PREFIX*: byte = 1


etan-status · 2021-10-06T10:31:06Z

This extended config is now properly validated by Lodestar's light client demo site https://github.com/ChainSafe/eth2-light-client-demo/tree/dapplion/update-to-prater after adjusting it to handle CONSTANT_CASE in /config/spec. Will submit a PR to Lodestar regarding uppercase once LGPL CLA has been cleared by legal.

etan-status · 2021-10-06T10:35:19Z

beacon_chain/rpc/rpc_config_api.nim

+
+      # https://github.com/ethereum/consensus-specs/blob/v1.1.1/presets/mainnet/phase0.yaml
+      "MAX_COMMITTEES_PER_SLOT":
+        $MAX_COMMITTEES_PER_SLOT,


Not sure why for RPC we need $ and for REST we need Base10.toString. I kept it as is for now, out of scope of this PR.

json-rpc uses some random, but particular encoding that others have come to rely upon - we must make sure that we don't change the formatting in any way between releases.

rest has a standard-defined output format, so the rest implementation uses more precise definitions that correspond to that standard - ie $ is some random string format for a type

Ah, that makes sense, thanks!

etan-status · 2021-10-06T10:48:30Z

beacon_chain/validator_client/api.nim

  let configFlag =
-    info.MAX_VALIDATORS_PER_COMMITTEE != MAX_VALIDATORS_PER_COMMITTEE or


Here, it felt quite random which of the constants were being checked. I extended this to cover all the constants exchanged via the config REST/RPC API that are hard-coded and not overridable using a dynamic configuration in Nimbus.

Discussed with @cheatfate , have to be careful on these comparisons, as this breaks compatibility with other clients that do not include the new extra constants as part of their config response.

Reverting this file!

etan-status · 2021-10-06T13:00:47Z

Update regarding licensing: The light client component of Lodestar is Apache licensed. Only other parts of Lodestar are LGPL licensed, so going ahead and submitting PR there to fix their config/spec endpoint as well.

etan-status · 2021-10-06T13:03:01Z

ChainSafe/lodestar#3325

etan-status · 2021-10-06T13:41:58Z

JSON-RPC is expected to only return v1.0.0 constants. Updated the PR accordingly.

New RPC output:

% curl -d '{"jsonrpc":"2.0","method":"get_v1_config_spec","params":[],"id":1}' -H 'Content-Type: application/json' localhost:9190 -s | jq

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "CONFIG_NAME": "mainnet",
    "MAX_COMMITTEES_PER_SLOT": "64",
    "TARGET_COMMITTEE_SIZE": "128",
    "MAX_VALIDATORS_PER_COMMITTEE": "2048",
    "MIN_PER_EPOCH_CHURN_LIMIT": "4",
    "CHURN_LIMIT_QUOTIENT": "65536",
    "SHUFFLE_ROUND_COUNT": "90",
    "MIN_GENESIS_ACTIVE_VALIDATOR_COUNT": "16384",
    "MIN_GENESIS_TIME": "1614588812",
    "HYSTERESIS_QUOTIENT": "4",
    "HYSTERESIS_DOWNWARD_MULTIPLIER": "1",
    "HYSTERESIS_UPWARD_MULTIPLIER": "5",
    "SAFE_SLOTS_TO_UPDATE_JUSTIFIED": "8",
    "ETH1_FOLLOW_DISTANCE": "2048",
    "TARGET_AGGREGATORS_PER_COMMITTEE": "16",
    "RANDOM_SUBNETS_PER_VALIDATOR": "1",
    "EPOCHS_PER_RANDOM_SUBNET_SUBSCRIPTION": "256",
    "SECONDS_PER_ETH1_BLOCK": "14",
    "DEPOSIT_CHAIN_ID": "5",
    "DEPOSIT_NETWORK_ID": "5",
    "DEPOSIT_CONTRACT_ADDRESS": "0xff50ed3d0ec03ac01d4c79aad74928bff48a7b2b",
    "MIN_DEPOSIT_AMOUNT": "1000000000",
    "MAX_EFFECTIVE_BALANCE": "32000000000",
    "EJECTION_BALANCE": "16000000000",
    "EFFECTIVE_BALANCE_INCREMENT": "1000000000",
    "GENESIS_FORK_VERSION": "0x00001020",
    "BLS_WITHDRAWAL_PREFIX": "0x00",
    "GENESIS_DELAY": "1919188",
    "SECONDS_PER_SLOT": "12",
    "MIN_ATTESTATION_INCLUSION_DELAY": "1",
    "SLOTS_PER_EPOCH": "32",
    "MIN_SEED_LOOKAHEAD": "1",
    "MAX_SEED_LOOKAHEAD": "4",
    "EPOCHS_PER_ETH1_VOTING_PERIOD": "64",
    "SLOTS_PER_HISTORICAL_ROOT": "8192",
    "MIN_VALIDATOR_WITHDRAWABILITY_DELAY": "256",
    "SHARD_COMMITTEE_PERIOD": "256",
    "MIN_EPOCHS_TO_INACTIVITY_PENALTY": "4",
    "EPOCHS_PER_HISTORICAL_VECTOR": "65536",
    "EPOCHS_PER_SLASHINGS_VECTOR": "8192",
    "HISTORICAL_ROOTS_LIMIT": "16777216",
    "VALIDATOR_REGISTRY_LIMIT": "1099511627776",
    "BASE_REWARD_FACTOR": "64",
    "WHISTLEBLOWER_REWARD_QUOTIENT": "512",
    "PROPOSER_REWARD_QUOTIENT": "8",
    "INACTIVITY_PENALTY_QUOTIENT": "67108864",
    "MIN_SLASHING_PENALTY_QUOTIENT": "128",
    "PROPORTIONAL_SLASHING_MULTIPLIER": "1",
    "MAX_PROPOSER_SLASHINGS": "16",
    "MAX_ATTESTER_SLASHINGS": "2",
    "MAX_ATTESTATIONS": "128",
    "MAX_DEPOSITS": "16",
    "MAX_VOLUNTARY_EXITS": "16",
    "DOMAIN_BEACON_PROPOSER": "0x00000000",
    "DOMAIN_BEACON_ATTESTER": "0x01000000",
    "DOMAIN_RANDAO": "0x02000000",
    "DOMAIN_DEPOSIT": "0x03000000",
    "DOMAIN_VOLUNTARY_EXIT": "0x04000000",
    "DOMAIN_SELECTION_PROOF": "0x05000000",
    "DOMAIN_AGGREGATE_AND_PROOF": "0x06000000"
  }
}

etan-status · 2021-10-06T13:42:45Z

beacon_chain/rpc/rest_config_api.nim

@@ -20,24 +20,19 @@ proc installConfigApiHandlers*(router: var RestRouter, node: BeaconNode) =
    cachedConfigSpec =
      RestApiResponse.prepareJsonResponse(
        (
+          # https://github.com/ethereum/consensus-specs/blob/v1.0.0/configs/mainnet/phase0.yaml


This one no longer exists in newer spec versions. There is now a PRESET_BASE though. I guess we keep this around for backwards compatibility?

prysm also left this constant in, but on Prater, for them it's value is prater and in nimbus it's mainnet.

etan-status · 2021-10-06T13:58:45Z

Rebased on unstable after the other REST fixes were merged.

The additional constants introduced with this are:

ATTESTATION_SUBNET_COUNT
ETH1_ADDRESS_WITHDRAWAL_PREFIX
MERGE_FORK_EPOCH
MERGE_FORK_VERSION
SHARDING_FORK_EPOCH
SHARDING_FORK_VERSION
TERMINAL_BLOCK_HASH
TERMINAL_TOTAL_DIFFICULTY

etan-status · 2021-10-06T15:14:45Z

Added the constants from prysm as well now.

Now, compared to Prysm stable:

nimbus have extra:
- ATTESTATION_SUBNET_COUNT
- ETH1_ADDRESS_WITHDRAWAL_PREFIX
- TERMINAL_BLOCK_HASH
- TERMINAL_TOTAL_DIFFICULTY
prysm has extra:
- MIN_ANCHOR_POW_BLOCK_DIFFICULTY
- TRANSITION_TOTAL_DIFFICULTY

etan-status · 2021-10-06T15:16:33Z

beacon_chain/rpc/rest_config_api.nim

+          DOMAIN_CONTRIBUTION_AND_PROOF:
+            "0x" & ncrutils.toHex(
+              uint32(DOMAIN_CONTRIBUTION_AND_PROOF).toBytesLE()),
+          # PARTICIPATION_FLAG_WEIGHTS


also left out on prysm

etan-status · 2021-10-06T15:17:36Z

beacon_chain/spec/eth2_apis/eth2_rest_serialization.nim

+  let svalue = reader.readValue(string)
+  try:
+    value = parse(svalue, UInt256, 10)
+  except AssertionError, RangeError:


@mratsim This is the location with the unexpected AssertionError. There is a reachable path for that one when processing a string starting with 0x, 0o or 0b.

This has been fixed to use ValueError now.

etan-status · 2021-10-06T15:19:38Z

Wait for new stint to fix parse function. Throw Catchable exceptions when parsing nim-stint#109

etan-status · 2021-10-08T10:36:04Z

Depends on bump nim-stint #2969

zah · 2021-10-14T11:07:46Z

beacon_chain/spec/eth2_apis/eth2_rest_serialization.nim

+## UInt256
+proc writeValue*(w: var JsonWriter[RestJson], value: UInt256) {.
+     raises: [IOError, Defect].} =
+  writeValue(w, toString(value))


Have you verified that decimal encoding is appropriate? There are some traditions in the Ethereum world to use hexademically encoded big integers.

Spec:

Values are returned with following format:

any value starting with 0x in the spec is returned as a hex string

numeric values are returned as a quoted integer

It seems to not depend on "big integer" for the config/spec but instead on whether the spec defines it as 0x or not.

There is still the general problem that config/spec is underspecified, it's just a map of random constants.
Lodestar accepts it in the current form (their light client requires the server to provide it).
Prysm is still on the older TRANSITION_TOTAL_DIFFICULTY which was a smaller number than 256 bit.
I guess for now it's okay.

Also: ethereum/consensus-specs#2645

etan-status · 2021-10-15T08:14:44Z

Updated links for 1.1.2 as well (no changes upstream)

zah · 2021-10-18T11:43:38Z

@cheatfate, can you comment regarding how this breaks the validator client expectations?

cheatfate · 2021-10-18T11:51:40Z

Every client prysm, lighthouse, teku, infura could use different subset of configuration parameters. As soon as this PR will be merged our own validator_client will require all the fields to be present in the config response.
To fix this issue, we need to create some minimal configuration preset of values which is present in all beacon_nodes and use it for configuration requests made by validator_client.

etan-status · 2021-10-18T11:53:39Z

Some more context:

An earlier commit on this PR included additional changes to RPC and in validator client logic.

The RPC changes were unnecessary, as it is intended that RPC is stuck at v1.0 of spec. These changes were reverted, and replaced with a comment.
The VC changes were strictly incorrect, as they strengthened the spec compatibility checks, therefore breaking compatibility with older peers who do not include all the additional config fields. These changes were also reverted.

Regarding VC, these were the incorrect changes: 3999285#diff-0efa923f42e20f0ff6915fd060c9eb33ac1d997af5bd003871f22da721886a17

The only remaining changes here are additional returned constants in the REST API for compatibility with Lodestar light client demo.

cheatfate · 2021-10-18T12:00:25Z

Because nim-presto using strict serialization/deserialization it could require to build alternative client calls like it was done here
https://github.com/status-im/nimbus-eth2/blob/unstable/beacon_chain/spec/eth2_apis/rest_debug_calls.nim#L16-L67

And use requreAllFields = false while deserializing response.

arnetheduck · 2021-10-18T18:10:10Z

#2764 has the skip-unknown-stuff patch

etan-status · 2021-10-19T14:58:18Z

Thanks for the additional background on Presto. I was not aware of it erroring out when encountering unknown fields. Good catch.

I think that the fix from @arnetheduck may be sufficient to fix this problem for the spec response as well (although I'm not sure it is correct to blindly apply allowUnknownFields to all REST responses). I have played with a more restricted fix that only applies that attribute to the spec response and this worked fine.

Note that without having the proper requireAllFields default initializer (just 0?) that the reverse problem may also happen if a field is ever removed from the config response. So, either having default initializers for any field not strictly necessary is required, or the introduction of a sub-type that only spans the few fields that are actually relevant to the VC (and then requesting the spec for the sub-type with allowUnknownFields, instead of attempting to parse the full spec with requireAllFields).

The introduction of a new sub-type was also what @zah recommended to me via DM. I'll create a separate PR for that, and turn this one back to "draft" for now, until new spec fields can be safely introduced that are not yet advertised by all the major clients.

etan-status · 2021-10-20T10:15:24Z

#3005

etan-status · 2021-10-20T17:43:41Z

(updated links to 1.1.3)

etan-status · 2021-10-20T23:55:30Z

beacon_chain/rpc/rest_beacon_api.nim

+  ./rest_utils,
  ../beacon_node, ../networking/eth2_network,
  ../consensus_object_pools/[blockchain_dag, exit_pool, spec_cache],
  ../validators/validator_duties,
  ../spec/[eth2_merkleization, forks, network],
-  ../spec/datatypes/[phase0, altair],
-  ./rest_utils
+  ../spec/datatypes/[phase0, altair]


This re-ordering avoids a compilation error on i386 that occurs otherwise, ever since this PR has been rebased on top of #2997

etan-status · 2021-10-22T13:20:45Z

As of #3005, the validator client now uses RestSpecVC so should no longer be affected by any newly introduced fields from here.

github-actions · 2021-10-27T09:44:47Z

Unit Test Results

    12 files ±0   764 suites ±0 33m 56s ⏱️ + 2m 29s
1 540 tests ±0 1 524 ✔️ ±0 16 💤 ±0 0 ❌ ±0
9 256 runs ±0 9 192 ✔️ ±0 64 💤 ±0 0 ❌ ±0

Results for commit dae875a. ± Comparison against base commit d6cd1cd.

♻️ This comment has been updated with latest results.

So far, the REST config response did not include all spec constants. The spec for `/eth/v1/config/spec` defines that the response should include constants for all hard forks known by the beacon node. This patch extends the corresponding response to include more constants.

etan-status mentioned this pull request Oct 6, 2021

Number of REST fixes. #2961

Merged

etan-status requested a review from cheatfate October 6, 2021 10:18

etan-status commented Oct 6, 2021

View reviewed changes

beacon_chain/spec/eth2_apis/eth2_rest_serialization.nim Outdated Show resolved Hide resolved

etan-status commented Oct 6, 2021

View reviewed changes

beacon_chain/spec/presets.nim

BLS_WITHDRAWAL_PREFIX*: byte = 0

ETH1_ADDRESS_WITHDRAWAL_PREFIX*: byte = 1

Copy link

Contributor Author

etan-status Oct 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🙃

etan-status commented Oct 6, 2021

View reviewed changes

etan-status force-pushed the spec-rest branch from 5909e00 to 1970f4e Compare October 6, 2021 13:41

etan-status commented Oct 6, 2021

View reviewed changes

etan-status force-pushed the spec-rest branch from 1970f4e to 3999285 Compare October 6, 2021 13:53

etan-status force-pushed the spec-rest branch 2 times, most recently from 9aa9f5b to c973cc0 Compare October 6, 2021 15:11

mratsim mentioned this pull request Oct 6, 2021

Throw Catchable exceptions when parsing status-im/nim-stint#109

Closed

etan-status commented Oct 6, 2021

View reviewed changes

etan-status force-pushed the spec-rest branch from c973cc0 to 4946f94 Compare October 8, 2021 11:06

zah reviewed Oct 14, 2021

View reviewed changes

etan-status force-pushed the spec-rest branch 2 times, most recently from 645ce8a to 7a00fd4 Compare October 15, 2021 08:10

etan-status marked this pull request as ready for review October 15, 2021 08:10

etan-status force-pushed the spec-rest branch from 7a00fd4 to eb38543 Compare October 15, 2021 08:14

etan-status changed the title ~~extend REST/RPC config with more spec constants~~ extend REST config with more spec constants Oct 18, 2021

etan-status marked this pull request as draft October 19, 2021 14:58

etan-status force-pushed the spec-rest branch from eb38543 to de46e7b Compare October 20, 2021 17:42

etan-status force-pushed the spec-rest branch from de46e7b to 86e0ab8 Compare October 20, 2021 23:54

etan-status commented Oct 20, 2021

View reviewed changes

etan-status marked this pull request as ready for review October 21, 2021 00:04

etan-status force-pushed the spec-rest branch 2 times, most recently from 5d4dc50 to 531d80a Compare October 27, 2021 08:11

etan-status force-pushed the spec-rest branch from 531d80a to dae875a Compare November 1, 2021 15:41

cheatfate approved these changes Nov 2, 2021

View reviewed changes

arnetheduck merged commit 1943258 into status-im:unstable Nov 2, 2021

etan-status deleted the spec-rest branch November 2, 2021 21:47

etan-status mentioned this pull request Nov 12, 2021

Implement light client syncing #2337

Closed

henridf mentioned this pull request Nov 29, 2022

Types and scaffolding for EIP-4844 #4365

Merged

		BLS_WITHDRAWAL_PREFIX*: byte = 0
		ETH1_ADDRESS_WITHDRAWAL_PREFIX*: byte = 1

		let configFlag =
		info.MAX_VALIDATORS_PER_COMMITTEE != MAX_VALIDATORS_PER_COMMITTEE or

extend REST config with more spec constants #2962

extend REST config with more spec constants #2962

Conversation

etan-status commented Oct 6, 2021

etan-status commented Oct 6, 2021

etan-status commented Oct 6, 2021

etan-status commented Oct 6, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

etan-status commented Oct 6, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

etan-status commented Oct 6, 2021

etan-status commented Oct 6, 2021

etan-status commented Oct 6, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

etan-status commented Oct 6, 2021

etan-status commented Oct 6, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

etan-status commented Oct 6, 2021 • edited Loading

etan-status commented Oct 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

etan-status commented Oct 15, 2021

zah commented Oct 18, 2021

cheatfate commented Oct 18, 2021

etan-status commented Oct 18, 2021

cheatfate commented Oct 18, 2021

arnetheduck commented Oct 18, 2021

etan-status commented Oct 19, 2021

etan-status commented Oct 20, 2021

etan-status commented Oct 20, 2021

Choose a reason for hiding this comment

etan-status commented Oct 22, 2021

github-actions bot commented Oct 27, 2021 • edited Loading

Unit Test Results

etan-status commented Oct 6, 2021 •

edited

Loading

etan-status commented Oct 8, 2021 •

edited

Loading

github-actions bot commented Oct 27, 2021 •

edited

Loading