[FR] Configuration reload without restart

[poupas]

This request is related to point 5. of #3088. It aims to decrease downtime when adding new validator keys. Requiring Nimbus to do a full restart to pick-up new validator keys may become problematic for people running many validators.

We think this proposal complements #3088, but does not replace it.

A possible approach to reduce validation downtime is to implement a "reload" mechanism that simply reloads validator keys whenever a signal (typically SIGHUP) is received. This pattern is fairly common for long-running daemons. Some examples:

• OpenSSH
• Docker
• Apache
• nginx
• MySQL

Another possible strategy is to listen for filesystem changes on specific files/directories (e.g. the validator keys directory), using a mechanism such as inotify. Whenever changes are detected, Nimbus could reload the keys. Some daemons that implement a similar strategy:

• Envoy
• uwsgi

This second approach might be superior to sending a signal as it may work better in a multi-container scenario. For example, imagine we have two containers: control_container and nimbus_container (where Nimbus is running). If control_container wants to notify Nimbus to restart, it would require that both control_container and nimbus_container share the same PID namespace.

Yet another approach is to listen for commands on a UNIX socket used exclusively for administrative tasks (e.g. reload, restart, etc.)

[zah]

Version 1.6.0 adds support for the standardized Keymanager API that can be used to add, remove or migrate validators on the fly.

An easy-to-use UI for validator management that will eventually support all clients is currently being developed here:
https://github.com/joaquim-verges/eth2-keymanager-frontend