Feature Request: simpler and safer workflow to move validators from one lighthouse vc to another

[winksaville]

Description

There should be a workflow to move one or more validators from one lighthouse vc to another. This workflow should minimize missed attestations, ideally zero, and should make all best attempts to guarantee no slashing.

Version

v1.5.1-stable

Present Behavior

Assume I have two lighthouse vc instances running on the same or different computers, VC1 and VC2. Each with a different set of enabled validators in their respective validator_definitions.yml file. I want to move a subset of the validators from VC1 to VC2. To do so I have to perform the following steps:

1. Stop VC1.
2. Edit VC1 valdiator_definitions.yml and disable all validators that are to be moved to VC2
3. Export VC1 slashing protection data.
4. Start VC1.
5. Start count down timer for 2 epochs (approx 13 min)
6. Stop VC2.
7. If the validators from VC1 do not exist in VC2 validator_definitions.yml then import them to VC2.
8. Edit VC2 validator_definitions.yml and ENABLE the validators that are to be moved from VC1 to VC2 and leave all current VC2 validators enabled.
9. Import the slashing protection data exported in step 3 to VC2.
10. Edit VC2 validator_definitions.yml and DISABLE only the validators that are to be moved from VC1 to VC2.
11. Start VC2.
12. Edit VC2 validator_definitions.yml and ENABLE the validators that are to be moved from VC1 to VC2.
13. Wait until timer from step 5 has counted down to zero.
14. Restart VC2 (i.e. stop VC2 and then start VC2).
15. The validators will now be running on VC2 and if no mistakes were made in the above steps "no slashings** will occur. But there will have been a number of missed attestations that would be unnecessary in an ideal case.

Expected Behavior

Ideally, the necessary steps should be able to be accomplished by issuing a single command specifying source VC, destination VC and a list of validators to move. That command should be able handle all details as specified above and atomically move the validators without user intervention. Plus, the there should be no missed attestations or slashings.

If the ideal case is not possible than the next best solution is that the only missed attestations would be for those validators that actually moved and only a minimal possibility of slashings.

Steps to resolve

I'm sure there is a non-trivial amount of work to do, but based on feedback from a question I posted on discord, https://discord.com/channels/605577013327167508/710978433823277099/882429074436407357, @michaelsproul suggests it might be possible.

[michaelsproul]

I think this is a great idea and a good candidate for inclusion in the standard validator client API being developed (ethereum/beacon-APIs#151).

On the implementation side, there are a few things that spring to mind:

1. We should fix issues with false positives on import: Import slash protection without checking slashability #2419. I'll start on that this week.
2. We need to restructure the slashing protection database internals so that it supports exporting a subset of validators.
3. We should spec out the /import and /export end points, describing the structure of requests/responses and the guarantees e.g. that export should atomically disable the selected validators and return a JSON object with keystores and EIP-3076 slashing protection data.
4. We could add a prototype implementation under the /lighthouse/ prefix until our proposal is accepted into the standard API

[winksaville]

A short discussion on discord it was mentioned an API might look something like:

struct ValidtatorInfo {
  ValidatorPubkey, // Maybe Kestore has ValidatorPubkey ???
  Keystore,
  Password,
  Interchange,
  ...
}

stopValidators(vlist: Vec<ValidatorPubkey>) -> Result<Vec<ValidatorInfo>, Error>;
startValidators(Vec<ValidatorInfo>) -> Vec<(ValidatorPubkey, Result<(), Error>)>;

Note; the Vec<ValidatorInfo> passed to startValidators is not necessarily the same as returned by stopValidators as the entity moving validators may be moving them among a set of Validator Clients.

[michaelsproul]

A standard API has been specced by Dapplion that would be suitable for our use-case.

As noted in my review, it is slightly different from the original vision described here in that it does not support exporting the keystores or extracting them from the validator. I think we can still make the atomic migrate work by having the CLI coordinator read the keystores from disk (possibly from the Lighthouse datadir, or elsewhere).

[winksaville]

My knee jerk reaction is that we can support the standard API but create a private API that does the complete job.

Your point about security is very prudent, I'd suggest that all of API's require a separate PKI keypair to be used to establish a connection. Of course a user can always use an SSH tunnel, but I think having it built-in would be preferred.

Note: These are the opinion of a neophyte :)