Validator does not submit attestations right after chain head comes

[twoeths]

Describe the bug

Chain head comes on time, however sometimes validator does not submit attestation right after that

Beacon node log

Apr-12 05:35:24.013[CHAIN]         ^[[36mverbose^[[39m: Clock slot slot=2768877
Apr-12 05:35:24.703[NETWORK]       ^[[36mverbose^[[39m: Received gossip block slot=2768877, root=0x0dab…b505, curentSlot=2768877, peerId=bafzaajiiaijccavvvmsdwytnua7vrs4dwhpkdppappey6u56y3yj4t6y7fibb46eim
Apr-12 05:35:25.277[CHAIN]         ^[[36mverbose^[[39m: Block processed slot=2768877, root=0x0dab0d1349355fee573c1308daf1f8ee5c1808ba246daa429207c766556fb505
Apr-12 05:35:31.658[API]             ^[[34mdebug^[[39m: Req req-m8 127.0.0.1 GET:/eth/v1/validator/attestation_data

Validator log

Apr-12 05:35:25.284[]              ^[[36mverbose^[[39m: Found new chain head slot=2768877, head=0x0dab0d1349355fee573c1308daf1f8ee5c1808ba246daa429207c766556fb505, previouDuty=0xb8f6e435cc4b738fa003a6518a63621b658783bddce157deb38c3994a002f546, currentDuty=0x49a8b9909c32af3c424c0e321f8838d2793ce49b3ac097fcf74e1b698f4d4488
Apr-12 05:35:31.664[]                ^[[34mdebug^[[39m: Signed attestation slot=2768877, index=4, head=0x0dab0d1349355fee573c1308daf1f8ee5c1808ba246daa429207c766556fb505, validatorIndex=200336

This was found even after the fix of #3906

Expected behavior

Validator should submit attestation right after it sees new head

[twoeths]

it looks like this is the beacon node's I/O lag issue, not the validator issue:

Apr-12 06:15:01.948[]              ^[[36mverbose^[[39m: Found new chain head slot=2769075, head=0x4037009ea6dccfa854e2141bfa7aa4b4305d1d6614632475d79c09f49625381d, previouDuty=0x71b160050d26e63fe913fe3b090a9677ac0bc33139e1557e7a3d9dd4f160700b, currentDuty=0x60ea055172b06f1766fe4da73475b0ce5c178c211cfd7818ef87e59b3c17df2e
Apr-12 06:15:01.949[]                 ^[[32minfo^[[39m: @@@ produceAndPublishAttestations slot=2769075, index=5
Apr-12 06:15:04.114[]                ^[[34mdebug^[[39m: Signed attestation slot=2769075, index=5, head=0x4037009ea6dccfa854e2141bfa7aa4b4305d1d6614632475d79c09f49625381d, validatorIndex=200340
Apr-12 06:15:11.558[]                 ^[[32minfo^[[39m: Published attestations slot=2769075, index=5, count=1

it's 2s delay until the node can process the attestation_data request, and 3s delay until the attestation is published

Apr-12 06:15:04.098[API]             ^[[34mdebug^[[39m: Req req-1b4 127.0.0.1 GET:/eth/v1/validator/attestation_data
Apr-12 06:15:07.060[API]             ^[[34mdebug^[[39m: Req req-1b5 127.0.0.1 POST:/eth/v1/beacon/pool/attestations

[twoeths]

root cause is #4002

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.