Replace committee exponential backoff with max progress

[vbuterin]

Removes the mechanism that only rotates committees if blocks have been finalized and every shard has been crosslinked or at exponentially decreasing intervals, and replaces it with a rule that shard committees can only progress a maximum of 64 epochs at a time to preserve the invariant that maximum possible work required per epoch for a validator is O(1).

This also fixes a bug in process_crosslink regarding the use of previous_epoch.

[JustinDrake]

👍

[djrtwo]

I don't think you can just get rid of previous and current_shuffling_epoch. In the case of non-finality, your v-set needs to remain stable against some epoch in the past and we need to remember two epochs worth of the shuffling epoch.

[vbuterin]

I don't think you can just get rid of previous and current_shuffling_epoch. In the case of non-finality, your v-set needs to remain stable against some epoch in the past and we need to remember two epochs worth of the shuffling epoch.

Your validator set needs to stay stable (at least until I write the withdrawal queue -> exit queue change), but the shuffling does not.

[djrtwo]

Your validator set needs to stay stable (at least until I write the withdrawal queue -> exit queue change), but the shuffling does not.

Right, but your validator set won't stay stable in this method. get_shuffling pulls the active validator set from the epoch passed in. Validator activations also have a 4 epoch delay. So if the epoch that you are getting your v-set from keeps incrementing even without finality, then your v-set can change without finality.

EDIT: You are also going to run into issues doing the randomness dynamically. If there are short range forks, then your shuffling could be different which would lead to unslashable attestations across forks.

[vbuterin]

EDIT: You are also going to run into issues doing the randomness dynamically. If there are short range forks, then your shuffling could be different which would lead to unslashable attestations across forks.

This is why slashing uses SlashableAttestations (possibly being renamed StandaloneAttestations) instead of using regular attestations; that solves this problem. Note that the need to use SlashableAttestations is unavoidable anyway because even in the current system, we reshuffle with exponential backoff even if we don't finalize, and you can get finality breaks from equivocations between two different epochs.

Shuffling is relevant only to crosslinks and slot selection; it has near-zero relevance to the Casper finalization mechanism, which treats all active validators in each epoch identically (not exactly zero because if you're close to the end of an epoch it's more likely your attestation will have to wait an extra epoch to be processed, but that's a small detail...) So I think it's good to treat (i) justification/finality and (ii) shuffling as completely separate mechanics.