Validator client subscription to slashing SSE events

[philknows]

Problem description

With the implementation of #6209 to detect slashable events, it still requires either useful infrastructure/3rd party software to take advantage of this feature. An idea is to further extend this into our validator client to benefit from this was brought up by @g11tech.

Solution description

By implementing a feature for alerting VCs of the signed messages, or having the connected validator to subscribe to these events, we can have the validator client itself be aware that one of their local pubkeys is being slashed. Then, we can implement some action (e.g. shutting down the service) to prevent further slashings.

Additional context

No response

[philknows]

Could I give this one to you @nflaig when you're back? This will be useful for us on infrastructure side and anybody else that's running a large number of keys with Lodestar.

[nflaig]

Then, we can implement some action (e.g. shutting down the service) to prevent further slashings.

I am wondering what the correct measure here is because shutting down the VC might not have any effect if people are running it via docker or systemd as it automatically restarts the process and I haven't found a specific exit code we could use to prevent this.

Possible options are

• shutdown process (as suggested)
• remove all validators keys from validator store (in-memory only)
• remove all validators from validator store and if imported, delete keystores from disk
• possibly implement it on the beacon node side instead, e.g. node stops submitting anything to the network

This will be useful for us on infrastructure side and anybody else that's running a large number of keys with Lodestar.

I am not sure this will be as useful if you are running multiple VC instances as we do as the validator can only keep track of it's own imported keys. A more large scale solution would be a external software that tracks and acts accordingly on all instances.

It really depends what kind of operator error we are expecting that could cause a slashing, will it affect only one VC or potentially all of them?

In any case, the proposed solution of this issue will be valuable for people running just a single VC instance and could still limit the damage caused even if multiple VCs are used.

[nflaig]

Also to since this is related to the topic, a way better approach to prevent slashing is by enabling doppelganger protection on every VC as most if not all slashings so far are related to running a second VC instance with the same keys.
And since we implemented #6012 there is basically no downside enabling doppelganger protection other than the first time a key is used it has to wait the 2 epochs.

[philknows]

I discussed this with @nflaig and came to the conclusion that internally for ChainSafe, we can rely on proactive measures such as doppelgänger protection to suit our needs. This is a bit lower on priority now given that there isn't a signal from other Lodestar users that this would be very useful for their setup in some way.

[wemeetagain]

Closing this issue. We will watch slashing events from external tooling and not add this to our validator client because we can't account for all setups.

[nflaig]

I would like to hear from Operators if this feature would be considered useful, I think it's not that hard to implement and best solution it likely to just remove the keys from in-memory store and log a warning + a metric that warns the operator. At least from discussion we had with Luca which developed Vero, this feature seems like something operators want.

Re-opening for now