Nimbus CL < > Lodestar VC incompatibility

[barnabasbusa]

Describe the bug

We are in the process to test cross beacon <> validator client compatibility, and found a bug when testing nimbus CL with lodestar VC.

Lodestar reports:

Apr-05 13:41:53.163[]                error: Error proposing block slot=6, validator=0x889f…366b - JSON expected key slot is undefined - Failed to produce block
Error: JSON expected key slot is undefined - Failed to produce block
    at ContainerType.fromJson (/usr/app/node_modules/@chainsafe/ssz/src/type/container.ts:347:15)
    at Object.fromJson (file:///usr/app/packages/api/src/utils/types.ts:156:44)
    at Object.fromJson (file:///usr/app/packages/api/src/utils/types.ts:198:15)
    at Object.fromJson (file:///usr/app/packages/api/src/beacon/routes/validator.ts:778:91)
    at Object.request [as produceBlockV3] (file:///usr/app/packages/api/src/utils/client/client.ts:91:50)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at BlockProposingService.produceBlockWrapper (file:///usr/app/packages/validator/src/services/block.ts:205:17)
    at BlockProposingService.createAndPublishBlock (file:///usr/app/packages/validator/src/services/block.ts:144:29)
    at async Promise.all (index 0)

Nimbus reports:

nothing out of the ordinary

Snooper between cl <> vc reports:

nothing out of the ordinary

Operating system

Linux

Nimbus version or commit hash

statusim/nimbus-eth2:multiarch-latest - sha256:8e79e02b37351bdcf0a4b07c9cb5a8dd23afdf83144a0d948644b72884cdd10d

Expected behavior

I would expect all client combinations would work.

Steps to reproduce

config.yaml:

participants:
  - cl_type: nimbus
    vc_type: lodestar
    vc_image: chainsafe/lodestar:next
additional_services:
  - dora
snooper_enabled: true
global_log_level: debug

kurtosis run github.com/kurtosis-tech/ethereum-package --args-file config.yaml

Additional context

Current BN <> VC Compatibility list tracker

Operating system

Linux

Lodestar version or commit hash

Version: v1.17.0/898cd90

cc: @pk910

[nflaig]

Thanks for all the detailed issues, this is great! This specific issue should be fixed by #6576 already (not yet released).

I am not exactly sure which commit hash you are running, it looks like docker hub does not provide a way to query the digest of an older version of a tag. I am also not able to pull it via digest anymore.

The commit hash can be printed out by running

docker run --rm <lodestar_image> --help

[barnabasbusa]

docker run --rm -it chainsafe/lodestar:next --help
🌟 Lodestar: TypeScript Implementation of the Ethereum Consensus Beacon Chain.

 * Version: v1.17.0/898cd90
  * by ChainSafe Systems, 2018-2024

[nflaig]

docker run --rm -it chainsafe/lodestar:next --help
🌟 Lodestar: TypeScript Implementation of the Ethereum Consensus Beacon Chain.

 * Version: v1.17.0/898cd90
  * by ChainSafe Systems, 2018-2024

The digest of this image is sha256:8d1ead7f7e0fbabb0431f4fb751c76128be0542701e89ab7b0b3719f82ac101f (see docker hub) which is different from what was referenced in the issue.

[barnabasbusa]

I must have had an older image locally stored for next. My bad! Nevertheless the issue persists, its just another error.

Apr-05 13:41:53.163[]                error: Error proposing block slot=6, validator=0x889f…366b - JSON expected key slot is undefined - Failed to produce block
Error: JSON expected key slot is undefined - Failed to produce block
    at ContainerType.fromJson (/usr/app/node_modules/@chainsafe/ssz/src/type/container.ts:347:15)
    at Object.fromJson (file:///usr/app/packages/api/src/utils/types.ts:156:44)
    at Object.fromJson (file:///usr/app/packages/api/src/utils/types.ts:198:15)
    at Object.fromJson (file:///usr/app/packages/api/src/beacon/routes/validator.ts:778:91)
    at Object.request [as produceBlockV3] (file:///usr/app/packages/api/src/utils/client/client.ts:91:50)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at BlockProposingService.produceBlockWrapper (file:///usr/app/packages/validator/src/services/block.ts:205:17)
    at BlockProposingService.createAndPublishBlock (file:///usr/app/packages/validator/src/services/block.ts:144:29)
    at async Promise.all (index 0)

[nflaig]

I must have had an older image locally stored for next

Based on the Nimbus log from #6634

agent_version=Lodestar/v1.17.0/def26ac node=http://172.16.0.28:8562[Lodestar/v1.17.0/def26ac] node_index=0 node_roles=AGBSDT

The commit the referenced image is build from is def26ac which did not include the skip_randao_verification fix yet, so this seems to be resolved at least.

its just another error.

That's a strange one, needs further investigation

[nflaig]

I must have had an older image locally stored for next. My bad! Nevertheless the issue persists, its just another error.

Apr-05 13:41:53.163[]                error: Error proposing block slot=6, validator=0x889f…366b - JSON expected key slot is undefined - Failed to produce block
Error: JSON expected key slot is undefined - Failed to produce block
    at ContainerType.fromJson (/usr/app/node_modules/@chainsafe/ssz/src/type/container.ts:347:15)
    at Object.fromJson (file:///usr/app/packages/api/src/utils/types.ts:156:44)
    at Object.fromJson (file:///usr/app/packages/api/src/utils/types.ts:198:15)
    at Object.fromJson (file:///usr/app/packages/api/src/beacon/routes/validator.ts:778:91)
    at Object.request [as produceBlockV3] (file:///usr/app/packages/api/src/utils/client/client.ts:91:50)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at BlockProposingService.produceBlockWrapper (file:///usr/app/packages/validator/src/services/block.ts:205:17)
    at BlockProposingService.createAndPublishBlock (file:///usr/app/packages/validator/src/services/block.ts:144:29)
    at async Promise.all (index 0)

This error seems to happen during deserialization of the response

lodestar/packages/api/src/beacon/routes/validator.ts

Line 778 in 898cd90

return {executionPayloadBlinded, executionPayloadSource, ...produceBlindedBlock.fromJson(data)};

It's a bit strange that we try to parse it as a BlindedBeaconBlock. I am assuming the blocks are full blocks and we actually receive BlockContents (post deneb).

A possible reason for this could be that Nimbus is sending execution_payload_blinded as a string value instead of boolean as per spec.

[nflaig]

Fixed by status-im/nimbus-eth2#6204