Global sidecar state for safe shutdown

[mempirate]

Context

Currently, there's no way to gracefully shut down the sidecar without risking a commitment fault. We should have a global, cloneable state watcher with the following states:

• Available
• PendingShutdown
• Shutdown

Upon a first interrupt (CTRL+C), the state should be set to PendingShutdown (should be event based, like a watch channel). This event will be caught by all actor tasks, and will kick off a process that checks whether we have any upcoming proposer duties with commitments. If we do, this process will wait until that slot has passed, and only then fire a Shutdown event, which will result in the actual shutting down of all actors and finally the process itself.

The API should ALWAYS reflect the state in any API responses (503 Service Unavailable if PendingShutdown). This ensures we can actually shut down without adding in new commitments.

So we should have a bi-directional communication channel:

• top level interrupt handler fires an event to all receivers
• receivers start cleaning up, and send an ack back up the channel when done
• only shut down when all acks are received

Note that this would be slightly less important with a persistent fallback block cache, but still important.

We should also have a way to force exit (i.e. press ctrl+c 3 times or something)

[thedevbirb]

Slightly related: #552 (comment) (cancellation tokens)