Doppelganger false positives

[nflaig]

Describe the bug

Doppelganger protection produces false positives if validator publishes attestation in current epoch N with an inclusion delay/(or duplicate inclusion) in epoch N + 1.

This is due to the fact that liveness of validator is checked based on attestation seen in block attesters cache

lodestar/packages/beacon-node/src/chain/chain.ts

Line 315 in 61c838b

this.seenBlockAttesters.isKnown(epoch, index) ||

which is set when importing block

lodestar/packages/beacon-node/src/chain/blocks/importBlock.ts

Line 137 in 61c838b

this.seenBlockAttesters.addIndices(blockEpoch, indexedAttestation.attestingIndices);

This is expected to happen because the doppelganger service set epoch N + 1 as the next epoch to check

lodestar/packages/validator/src/services/doppelgangerService.ts

Line 69 in 61c838b

nextEpochToCheck: this.clock.currentEpoch + 1,

There are multiple factors which increase the chance to produce false positives

• validator client is restarted towards the end of epoch
• increased inclusion delay due to bad network conditions (e.g. on goerli)
• inaccurate source, target, or head vote of last attestation?
• number of validator keys

This is issue can be fully mitigated by setting next epoch to check to N + 2 at the cost of missing more validator duties.

Expected behavior

Doppelganger protection must not produce false positives.

Steps to reproduce

Enable doppelganger protection by setting --doppelgangerProtectionEnabled and restart validator client near the end of epoch (slot 28-31), it is important to make sure it is still started in the same epoch.

Additional context

Examples of false positives on goerli

Stopped validator client end of epoch 184455 (slot=5902590) and started it in same epoch (slot=5902591)

Started doppelganger protection

Jun-21 09:18:17.474[]                 info: doppelganger protection enabled currentEpoch=184455, detectionEpochs=1

Registration of validators

Jun-21 09:18:17.886[]                 info: Registered validator for doppelganger remainingEpochs=1, nextEpochToCheck=184456

Liveness response that triggered detection

Jun-21 09:24:46.559[]                debug: doppelganger pollLiveness currentEpoch=184456, indicesCount=1962
Jun-21 09:24:46.662[]                debug: getLiveness response epoch=184456

Doppelganger error

Jun-21 09:24:46.664[]                error: Doppelganger(s) detected
        A doppelganger occurs when two different validator clients run the same public key.
        This validator client detected another instance of a local validator on the network
        and is shutting down to prevent potential slashable offenses. Ensure that you are not
        running a duplicate or overlapping validator client 461176, 462364, 460910, 462319, 462359, 460903, 461460, 461144, 461632, 462398, 461023, 461221, 461013, 461130, 461381, 461578, 462233, 460645, 460598, 461119, 462390, 461539, 462379, 461561, 460995, 460566, 460825, 461309, 462318, 460682, 460782, 461827, 461278, 461584, 461873, 461615, 460989, 460941, 462324, 461145, 460700, 461866, 462067, 461452, 461667, 460828, 461485, 461394, 462471, 461922, 462332, 460546, 460776, 461551, 461181, 461966, 461963, 461115, 461370, 461128, 461945, 461228, 461155, 460597, 461838, 461082, 461157, 461372, 462056, 462146, 460699, 462249, 461759, 461494, 460554, 461096, 461151, 460617, 462317, 461347, 460850, 460927, 461995, 461266, 462196, 461336, 462171, 460908, 462182, 462189, 461572, 460586, 461087, 460970, 460897, 460860, 461543, 461733, 460525, 461826, 461272, 461771, 462232, 461689
Jun-21 09:24:46.665[]                error: Process shutdown requested  Doppelganger(s) detected
Error: Doppelganger(s) detected
    at DoppelgangerService.detectDoppelganger (file:///home/devops/goerli/lodestar/packages/validator/src/services/doppelgangerService.ts:209:36)
    at DoppelgangerService.pollLiveness (file:///home/devops/goerli/lodestar/packages/validator/src/services/doppelgangerService.ts:139:10)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Clock.runAtMostEvery (file:///home/devops/goerli/lodestar/packages/validator/src/util/clock.ts:96:7

---

Validator 461381

The attestation of epoch N was included multiple times in epoch N and N + 1 causing the false positive.

Last signed attestation

Jun-21 09:15:52.030[]                debug: Signed attestation slot=5902579, index=13, head=0x616b4a8f5af67af88f91bb2010365c64bcf793d9ac77d4456cf7fd49f2402e14, validatorIndex=461381

Registration of validator

Jun-21 09:18:17.580[]                 info: Registered validator for doppelganger remainingEpochs=1, nextEpochToCheck=184456, pubkeyHex=0x8a1429569307ada150e14f1c0971d6e418bbcc7f8aafdf1ceb81f487caefbf3109463e563b02459dc4f7a76e024c0e02

validatorSeenAtEpoch response on beacon node

Jun-21 09:24:46.620[chain]           debug: validatorSeenAtEpoch index=461381, epoch=184456, seenBlockAttesters=true, seenAttesters=false, seenAggregators=false, seenBlockProposers=false

Based on data from beaconcha.in there has not been an inclusion delay

However, the attestation is included in multiple blocks, first in slot 5902580 which is epoch 184455 but also in slot 5902593 which is epoch 184456. The 2nd inclusion likely caused seenBlockAttesters=true.

---

Validator 461016

This is from a another test run, it is expected for this validator that doppelganger protection detected a false positive as attestation inclusion was delayed to next epoch.

Last signed attestation epoch=184287

Jun-20 15:22:14.842[]                debug: Signed attestation slot=5897211, index=41, head=0x77443f93430711ab246de346bccbbf8d230089b79e76a399ef7c173878ac6673, validatorIndex=46101

Registered validator on epoch=184287

Jun-20 15:23:06.515[]                 info: Registered validator for doppelganger remainingEpochs=1, pubkeyHex=0x8186135e9b5cbba537b93a7a79a615e55b841e61abda623f28de0bcd893a7cf8285e808f48376fc6f97f600e57d65af

Last attestation epoch=184287, slot=5897211, inclusion_slot=5897222 (epoch=184288)

---

More examples can be found here: doppelganger-false-positives.md

Operating system

Linux

Lodestar version or commit hash

unstable (f677434)

[nflaig]

• Closing as this is effectively resolved by feat: restart aware doppelganger protection #6012

The suggested solution to set the next epoch to check to N + 2 won't be required to prevent false positives. The only way to reproduce this issue previously was by restarting the validator client close to the end of the current epoch but in this case we now skip doppelganger protection.