Leader continued replying with ReplicationFailed

[xirc]

Situation

The following log continued (up to 1700) in some fault injection tests:

[Leader] failed to replicate the event (type=[lerna.akka.entityreplication.raft.model.NoOp$]) since the entity (entityId=[0000059981], instanceId=[34156], lastAppliedIndex=[814]) must apply [1] entries to itself. The leader will replicate a new event after the entity applies these [1] non-applied entries to itself.

By diagnosing logs, the following situation happened:

1. RaftActor (replica-group-1) was the leader.
2. Entity (id=0000059981, replica-group-1) succeeded in replication of NoOp.
   • The entity's lastAppliedLogEntryIndex was 814.
   • The NoOp replication was succeeded with index 821.
3. RaftActor (replica-group-2, Follower) updated indices to 821 (commitIndex=821, lastApplied=821).
   • RaftActor (replica-group-2, Follower) didn't send Replica for index 821 to the entity since an associated event is NoOp.
   • Entity (id=0000059981, replica-group-2) didn't update its lastAppliedLogEntryIndex to 821.
4. RaftActor (replica-group-2) became the leader for some reasons.
5. Entity (id=0000059981, replica-group-2) received ProcessCommand and then attempted to replicate an event:
   • Entity (id=0000059981, replica-group-2) sent Replicate(entityLastAppliedIndex=814, ...)
6. RaftActor (replica-group-2, Leader) replied with ReplicationFaield

Replica for EntityEvent(Some(entityId), NoOp) is not sent:

akka-entity-replication/src/main/scala/lerna/akka/entityreplication/raft/RaftActor.scala

Lines 477 to 486 in 0bd7465

	def applyToReplicationActor(logEntry: LogEntry): Unit =
	logEntry.event match {
	case EntityEvent(_, NoOp) => // NoOp は replicationActor には関係ないので転送しない
	case EntityEvent(Some(entityId), event) =>
	if (log.isDebugEnabled) log.debug("=== [{}] applying {} to ReplicationActor ===", currentState, event)
	replicationActor(entityId) ! Replica(logEntry)
	case EntityEvent(None, event) =>
	if (log.isWarningEnabled)
	log.warning("=== [{}] {} was not applied, because it is not assigned any entity ===", currentState, event)
	}

Possible solutions

1. RaftActor will send Replica to an entity also if an EntityEvent contains NoOp.
2. Leader will start replication if non-applied entries contain only NoOp.