sync should not turn off consensus

[lrettig]

Description

I'm running two nodes (both v1.1.6), both see exactly the same layer data, but one thinks it's synced and one doesn't:

> grpcurl -plaintext localhost:9092 spacemesh.v1.NodeService.Status
{
  "status": {
    "connectedPeers": "44",
    "syncedLayer": {
      "number": 19386
    },
    "topLayer": {
      "number": 19395
    },
    "verifiedLayer": {
      "number": 19385
    }
  }
}
> grpcurl -plaintext localhost:9192 spacemesh.v1.NodeService.Status
{
  "status": {
    "connectedPeers": "1",
    "isSynced": true,
    "syncedLayer": {
      "number": 19386
    },
    "topLayer": {
      "number": 19395
    },
    "verifiedLayer": {
      "number": 19385
    }
  }
}

The one that thinks it's out of sync keeps printing:

2023-09-19T12:17:50.296-0400 INFO abcde.sync node is too far behind {"node_id": "abcde", "module": "sync", "sessionId": "06dadc41-5e8b-4f6a-8e67-7baff90bb12c", "current": "19395", "last synced": "19389", "behind threshold": 3, "name": "sync"}

The only difference between them is that the "in sync" node is a private, local peer connected only to the other node, and the "out of sync" node is a public node with 40-50 peers. Unclear if this is related.

[lrettig]

logfile for "out of sync" node:

log.txt.gz

[dshulyak]

i think this is a big problem. sync should be very conservative when it can interrupt consensus. in fact it should almost never interrupt it. current heuristic of 3 layers is no good, or something else makes it so that it doesn't work.
in latest outage it makes some nodes change sync state consistently.

[dshulyak]

lets change to not synced only if node was offline (0 peers reported by libp2p) for 30 minutes. in all other cases sync should not interrupt consensus

[lrettig]

looks like all private nodes are in sync while public ones aren't (https://discord.com/channels/623195163510046732/1141739980306272427/1153806007785496698, also: https://discord.com/channels/623195163510046732/1141739980306272427/1153911699691282482)

I noticed this behavior as well (before the patch, on v1.1.6). do we have any idea why this might've been the case?

[dshulyak]

i think thats because atx sync queries are slow, and it caused sync to be delayed and fall out of expected window. below you can see that problems stopped when epoch started, and atx sync stopped.

private query public and all queries are fast, but public queries random peers on the network and responses are unpredictable. i think this just highlighted that we should not have such risky heuristics.

[lrettig]

For the record this is what @tal-m said - in order for a node to participate in consensus:

It definitely needs to have timing information for the Hare rounds in the current instance, and IIRC for voting on proposals it needs information about the current state and the recent transactions in the mempool (to compute "conservative balance").
For generating ballots it needs information about the recent Hare instances, and if there are layers older than hdist that haven't passed the confidence threshold it also needs all the most recent ballots.
Of course, for doing anything it needs to be able to evaluate eligibility, so it needs to know the active set and any malfeasance proofs.

[dshulyak]

i was advocating for something similar in this topic #4504 (comment)

but i think think hare can have more relaxed conditions, unlike tortoise hare can't vote against, so it is the same as not voting at all. i think we will remove IsSynced condition all together, and rely on specific data being available.

but what is more important is that:

• sync can't stop consensus (maybe what i changed is enough, but i would remove this completely and turn off consensus only if i have 0 peers for longer than 20-30 minutes)
• additionally there is a persistent issue about selecting responsive peers, which is also quite important

[dshulyak]

likely same problem as #4977