v1.2.0: missed attestations due to insufficient peers

[twoeths]

Describe the bug

After we deploy v1.2.0-rc.1, there is a missed attestation due to insufficient peers:

validator-2022-10-15.log:7224:Oct-15 03:57:26.860[]                error: Error publishing attestations slot=4915185 Internal Server Error: PublishError.InsufficientPeers

The sent peers when we publish attestations are actually down compared to v1.1.0

Expected behavior

• Investigate why the sent peers are so low
• It should be the same to v1.1.0

[twoeths]

Number of topic peers are actually dropped.

• on a mainnet node with v1.1.0 then v1.2.0.rc-1 (8 validators)

• on a node with v1.1.0 only (8 validators)

[twoeths]

"Outgoing request error rate" is so high with v1.2.0

two of them are legacy errors like "stream ended before 1 bytes became available" or "REQUEST_ERROR_EMPTY_RESPONSE"

~/beacon$ grep -e "stream ended before 1 bytes became available\|REQUEST_ERROR_EMPTY_RESPONSE" -rn beacon-2022-10-15.log | wc -l
26168
~/beacon$ grep -e "stream ended before 1 bytes became available\|REQUEST_ERROR_EMPTY_RESPONSE" -rn beacon-2022-10-16.log | wc -l
~/beacon$ grep -e "stream ended before 1 bytes became available\|REQUEST_ERROR_EMPTY_RESPONSE" -rn beacon-2022-10-14.log | wc -l
29500
~/beacon$ grep -e "stream ended before 1 bytes became available\|REQUEST_ERROR_EMPTY_RESPONSE" -rn beacon-2022-10-12.log | wc -l
288
~/beacon$ grep -e "stream ended before 1 bytes became available\|REQUEST_ERROR_EMPTY_RESPONSE" -rn beacon-2022-10-11.log | wc -l
746

[twoeths]

since lodestar peer scores are low, we send goodbye to a lot of peers

[twoeths]

we also received a lot of goodbye requests for different reasons

[twoeths]

our I/O was super busy, other nodes disconnected us due to the timeout and we got penalties

[twoeths]

it's still an issue in v1.2.0:

[twoeths]

haven't seen this issue since we deploy v1.2.0 (at least in the 39-validator node). Also the number of topic peers (sent peers) grew up after the 1st 2 days

[nazarhussain]

I am seeing lot of these error messages on the unstable branch during the sim tests.

Eph 4/4 1.635[node-2-cl-lodestar/api] �[31merror�[39m: Error on submitPoolAttestations [0] slot=36, index=2 PublishError.InsufficientPeers
Error: PublishError.InsufficientPeers
    at EventTarget.publish (file:///home/runner/work/lodestar/lodestar/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:1957:13)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at EventTarget.publishObject (file:///home/runner/work/lodestar/lodestar/packages/beacon-node/src/network/gossip/gossipsub.ts:171:20)
    at EventTarget.publishBeaconAttestation (file:///home/runner/work/lodestar/lodestar/packages/beacon-node/src/network/gossip/gossipsub.ts:213:12)
    at file:///home/runner/work/lodestar/lodestar/packages/beacon-node/src/api/impl/beacon/pool/index.ts:52:31
    at async Promise.all (index 0)
    at Object.submitPoolAttestations (file:///home/runner/work/lodestar/lodestar/packages/beacon-node/src/api/impl/beacon/pool/index.ts:46:7)
    at Object.handler (file:///home/runner/work/lodestar/lodestar/packages/api/src/utils/server/genericJsonServer.ts:41:23)

[dapplion]

@tuyennhv Would be good to take a look at the sim tests run to understand why network health is bad

[twoeths]

@nazarhussain @dapplion I opened #4764