Nimbus Validator Client incompatible with Lighthouse Beacon Node

[gg718]

Using Nimbus Validator Client v23.5.1, I'm seeing the follow error in my logs when attempting to use Lighthouse BNs:

ERR 2023-05-19 11:07:12.037+00:00 Beacon node provides unexpected response   node=http://23.49.153.222:5052[Lighthouse/v4.1.0-693886b/x86_64-linux] node_index=0 node_roles=AGBSD
ERR 2023-05-19 11:07:12.037+00:00 Beacon node provides unexpected response   node=http://123.11.104.27:5052[Lighthouse/v4.1.0-693886b/x86_64-linux] node_index=2 node_roles=AGBSD
ERR 2023-05-19 11:07:12.054+00:00 Beacon node provides unexpected response   node=http://88.126.246.104:5052[Lighthouse/v4.1.0-693886b/x86_64-linux] node_index=1 node_roles=AGBSD
WRN 2023-05-19 11:07:12.054+00:00 Unable to get head state's validator information reason="Beacon node reports unexpected status" service=duties_service

How can I debug this further / provide more information to help identify the issue?

[arnetheduck]

Can you run the validator client with --log-level=DEBUG and provide the logs around the errors?

[gg718]

@arnetheduck Note the ?id=<snip> in the first log line is a long comma-separated list containing 1024 validator public keys:

DBG 2023-05-24 12:38:37.145+00:00 Received REST response body from remote server remote=23.49.153.222:5052 request="/eth/v1/beacon/states/head/validators?id=<snip> contentType=<missing> size=0 connection=97
DBG 2023-05-24 12:38:37.146+00:00 Sending REST request to remote server      remote=23.49.153.222:5052 request=/eth/v1/validator/duties/attester/123456 http_method=POST
DBG 2023-05-24 12:38:37.146+00:00 Received REST response body from remote server remote=23.49.153.222:5052 request=/eth/v1/validator/duties/proposer/123457 contentType=application/json size=5109 connection=100
DBG 2023-05-24 12:38:37.146+00:00 Received REST response body from remote server remote=23.49.153.222:5052 request=/eth/v1/validator/duties/attester/123457 contentType=application/json size=9609 connection=99
DBG 2023-05-24 12:38:37.147+00:00 Failed to deserialize REST JSON data       err="<data>(1, 0) Unexpected token 'tkEof' in place of 'object start bracket'" data=
DBG 2023-05-24 12:38:37.147+00:00 Failed to deserialize REST JSON data       err="<data>(1, 0) Unexpected token 'tkEof' in place of 'object start bracket'" data=
DBG 2023-05-24 12:38:37.147+00:00 Received unexpected error response         response_code=414 node=http://23.49.153.222:5052[Lighthouse/v4.1.0-693886b/x86_64-linux] node_index=0 node_roles=AGBSD reason="Unable to decode error response: [Serialization error]" strategy=first request=getStateValidators
ERR 2023-05-24 12:38:37.147+00:00 Beacon node provides unexpected response   node=http://23.49.153.222:5052[Lighthouse/v4.1.0-693886b/x86_64-linux] node_index=0 node_roles=AGBSD
WRN 2023-05-24 12:38:37.147+00:00 Unable to get head state's validator information reason="Beacon node reports unexpected status" service=duties_service

[arnetheduck]

Ah, this appears to be a problem with the lighthouse beacon node places a limit on the URL length that's lower than ours - cc @michaelsproul does LH have an option to accept longer ones?

In the meantime, one option have to split the validators into several validator clients, each holding fewer keys.

We'll also look into making this limit configurable.

[michaelsproul]

Hmm I don't think we've got a CLI flag for the URL limit. I'll open an issue

[gg718]

Another consideration: according to https://ethereum.github.io/beacon-APIs/#/Beacon/getStateValidators, the list can also be provided using the validators indexes which would be significantly shorter in length than the public keys.

[arnetheduck]

the list can also be provided using the validators indexes which would be significantly shorter in length than the public keys.

we call getStateValidators to find out what indexes they have ;) the VC doesn't know on startup.

[nflaig]

According to the beacon API spec the array limit is supposed to be 30 items. In Lodestar we query this API in batches of 10 but there is plenty of downstream tooling (e.g. rocketpool smartnode) which also just sends one request with all pubkeys as comma-seperated values in query string.