Missed block proposal due to delayed publish in gossipsub

[twoeths]

Describe the bug

Beacon node published the block on time, we published 2s into the slot

2024-03-25 14:42:35.001 | Mar-25 07:42:35.000[chain]         verbose: Clock slot slot=8710711

2024-03-25 14:42:36.670 | Mar-25 07:42:36.670[chain]            info: Publishing assembled block blockRoot=0x2e8c2dbe34c551e418869dc673c3631aa29b0fb03de04476a0f0673ce71cc4ae, slot=8710711, source=engine

Mar-25 07:42:36.987[network]       ^[[36mverbose^[[39m: Publish to topic topic=/eth2/6a95a1a9/beacon_block/ssz_snappy, sentPeers=46

however all other nodes received the block ~5s into the slot and finally the block was reorged by the next proposer

2024-03-25 14:42:42.397    
Mar-25 07:42:42.397[network]         debug: Received gossip block slot=8710711, root=0x2e8c…c4ae, curentSlot=8710711, peerId=16Uiu2HAmG7vshks6RM7A8gLrsVYX7yKRpFFetNCPmiSBXYTwhZ9d, delaySec=5.275000095367432, pending=null, haveBlobs=0, expectedBlobs=0, recvToValLatency=1.6109998226165771, recvToValidation=2.121999979019165, validationTime=0.5110001564025879

that's clearly a delay at gossipsub side

Expected behavior

No delay when publishing messages in gossipsub

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.17.0

[twoeths]

It was analysed by lighthouse that flood publishing may contribute to this delay, we should experiment the same. See sigp/lighthouse#4383 and sigp/lighthouse#5357

[philknows]

Closing, fixed by #6755 and enabled by default due to good results on #6815