continuously send new activations and malfeasance proofs to peers

[dshulyak]

we are not syncing activations and malfeasence proofs reliably (#4831)

• activations and proofs should be indexed in memory by timestamp when they are received. we should limit index to one full epoch
• peers can ask for a set of objects that were received after certain timestamp
• it will do so continuously every 5-30 minutes. there should be some randomization, for example round is 5 minutes, but we request only from a subset of peers on every round
• it will store timestamps for each individual peer for last X known peers. can use local db for that
• for unknown peers we ask for all malfeasence proofs (timestamp = 0) and activations that could be created in this epoch (timestamp=epoch_start.timestamp())
• we should be using native libp2p streams instead of custom fetcher code, as the latter is not very good and needs to be replaced

this code should start working after we executed initial sync that tries to download data from the very beggining without additional logic.

[ivan4th]

Related: #3987 (comment)
Using technique from Range-Based Set Reconciliation by Aljoscha Meyer, which is used for things like iroh-sync, it should be possible to sync activations and malfeasance proofs across peers cheaply and often without the peers keeping track of each other.

[dshulyak]

i think we can do something simpler for malfeasance proofs, following the pattern in #5599.

we can ask 5-10 peers for all equivocating identities every 10-30 minutes. and then download full missing proofs. we don't expect this number to be significant, as it effectively cancels all rewards. the ultimate attack here is that if all identities will equivocate, and this is bounded by the number of atxs.

this will enable distributed verification significantly faster.

[dshulyak]

proposed protocol:

• on node startup check how long ago proofs were synced, if it was in a previous epoch or a quarter of epoch ago download all malfeasance identities and download missing proofs synchronously from N peers. N can be 2-3, just to avoid downloading data from not very well synced node
• otherwise spin up a background loop to download malfeasance identities every M minutes from N peers. and then sync missing malfeasance proofs for those identities. the interval and number of peers can be parametrized, but should be chosen such that overhead is negligible.

for example, we have ~600 malfeasance/canceled identities, if this number grows to 3000 then every query will have to download 10KB of data. if we ask 10 peers every 30 minutes (not concurrently), thats 55 bytes per second.

[ivan4th]

I think this approach should work quite well at least for a few years (unless the number of malfeasant identities grows for some unanticipated reason, which is unlikely)

[pigmej]

Can we then try to expedite that to the mainnet? It will make us able to use distributed verification.