Leader cannot reply to an entity with a ReplicationFailed message in some cases

[xirc]

Situation

1. An entity with a non-latest state (there is a Replica or ReplicationSucceeded in its mailbox) runs its command handler and will send a Replicate message to the leader.
   • This situation is possible and expected.
2. The leader cannot reply to the entity with a ReplicationFailed message in some cases.
   • The leader should reply to the entity with a ReplicationFaield message against a Replicate message sent based on a non-latest state.
   • Leader can reply with a ReplicationFaield message if LogEntry for the previous entity's replicate message is uncommitted.
   • Leader cannot reply with a ReplicationFaield message if LogEntry for the previous entity's replicate message is committed, but not applied to the entity. There is a subtle timing in which the entity sends a Replicate based on a non-latest state after the leader has sent the LogEntry to the entity.

Possible Solution

An Entity sends a Replicate message containing its lastAppliedLogEntryIndex. A leader will reply with ReplicaionFaield if Leader's Raft log (ReplicatedLog) contains an entry that satisfies the all of the following conditions:

• The entry is for the entity (LogEntry.event.entityId is equal to Replicate.entityId)
• The index of the entry is greater than the index (lastAppliedLogEntryIndex) of the Replicate message.