RaftActor should start automatically after a cluster startup

[xirc]

Suppose that the following scenario:

• Domain event A was replicated to all Raft Actors.
• Domain event A was already committed.
• CommitLogStore did not save the domain event A for some reason:
  • Persistent Store behind CommitLogStore was not available temporarily.
  • Cluster stopped before CommitLogStore saved the event.

After we conduct a full cluster restart, CommitLogStore will not save the domain event A until the cluster receives a user request against a RaftActor that should handle the domain event A. This is because RaftActor will not start until the cluster receives a user request. This behavior means we cannot subscribe to the domain event A on a query-side until the cluster receives a user request.

Expected

CommitLogStore will save domain events automatically after a cluster startup. This behavior should not require any user request. This save enables us to subscribe to the event. To achieve this behavior, RaftActors should start automatically after a cluster startup without any user requests.

[xirc]

We have some options to achieve this issue.

1. Use Remember Entities (https://doc.akka.io/docs/akka/2.6.18/typed/cluster-sharding.html#remembering-entities)
   A cluster can start all RaftActors that was running before a cluster stop since RaftActor is an entity of Cluster Sharding. This option requires us to use an additional persistent store since we must use Remember Entities with eventsourced mode.

2. Send ShardRegion.StartEntitys to ShardRegion
   A cluster can start all RaftActors since it can calculate all possible RaftActors entity ids (if we use Typed API). This option requires us to run all RaftActors which is not used yet. This restriction might not critical since a cluster runs all RaftActors eventually, and does not passivate it. This option requires no additional persistent store. This option cannot support classic APIs.

We will use option 2 based on team discussions.