An entity could stick at WaitForReplication when a Raft log entry is truncated by conflict

[xirc]

Situation

1. An entity on node A sends a Replicate message to its RaftActor (supposed that called as RaftActor A), and then waits for a Replication result (ReplicationSucceeded, ReplicationFaield, or Replica).
2. RaftActor A becomes a Follower for some reason.
   • There is another leader at this point (supposed that in node B).
   • Suppose that a Raft log entry for the replication (started by node A's Raft Actor) is conflicted for some reason, the new leader truncates the entry.
   • The replication for the entity will not progress; The old leader on node A will not send any Replication result to the entity.
3. Suppose that the leader on node B didn't receive a command for the entity. The entity's state is WaitForReplication since the entity doesn't receive any Replica.
4. RaftActor A becomes a Leader again for some reason.
5. The entity continues to stash new incoming ProcessCommand because its state is WaitForReplication.

Possible Solution

RaftActor sends a ReplicationFaied message to an entity if the entity is waiting for a Raft log entry truncated by conflict. This might be possible to achieve at AppendEntries handling:

• akka-entity-replication/src/main/scala/lerna/akka/entityreplication/raft/Leader.scala

  Line 99 in 0acd526

  val newEntries = currentData.resolveNewLogEntries(appendEntries.entries)

• akka-entity-replication/src/main/scala/lerna/akka/entityreplication/raft/Follower.scala

  Line 110 in 0acd526

  val newEntries = currentData.resolveNewLogEntries(appendEntries.entries)

• akka-entity-replication/src/main/scala/lerna/akka/entityreplication/raft/Candidate.scala

  Line 153 in 0acd526

  val newEntries = currentData.resolveNewLogEntries(appendEntries.entries)

Conflicted entries are existing entries of the Raft log (called ReplicatedLog) with indices greater than or equal to the index of the head of newEntries.