rfc: add leader-based journal

[w41ter]

The rendered version: leader based journal

[w41ter]

cc @huachaohuang

[zojw]

Confirm another question~ (storage seems also have the same question

Now, we have 2 engine nodes (L1 and F2), and we add 2 engine Nodes(F3, F4) to the cluster, how do F3, F4 know whether they need to replicate L1's data?

Or where is suitable to maintain metadata like default-tablet-group -> Leader L1, Follower F2, F4, Nothing F3? (ps. F3 maybe as an active follower in other tablet-group in future)

Orchestrator could give a full list of L1, F2, F3, F4, but it doesn't know F3 no need to replicate...

[w41ter]

@zojw Now we only need to implement single writer journal, it means that only one engine is writer, the others are readers. But once partition is added to engine, we should add a new node to manage the destribution of partitions.

[tisonkun]

@w41ter-l @huachaohuang I noticed that we're implementing the API designed here. If the RFC is accepted, please update the status and pull request links, start a tracking issue for the work (added in the RFC) and merge this PR.

Otherwise, the RFC process is fake.

[tisonkun]

Suggested change

	- Status: draft
	- Pull Request:
	- Status: accepted
	- Pull Request: https://github.com/engula/engula/pull/280
	- Tracking Issue: <please create one of this>

[tisonkun]

co https://engula.zulipchat.com/#narrow/stream/306467-general/topic/random/near/267827815

cc @huachaohuang @w41ter-l

[tisonkun]

Who is expected to perform follower read?

If it happens only during recovery, duplicate followers seems overkill.

[w41ter]

An engine follower must read the stream for the duration of its life to keep track of the leader's state. And no follower read is performed during recovery, because the recovery may execute in leader,

[tisonkun]

Thanks for your explanation :)

[huachaohuang]

I think this RFC proposes two things: a new SingleWriteJournal API and an implementation of the API. I suggest that we split the RFC into two, one RFC describes the semantic of SingleWriteJournal (I prefer LeaderBasedJournal, though), another RFC describes an implementation of the new API.

[tisonkun]

Here we have two focuses:

1. For persisting data into journal server cluster, the proposal is using a quorum-based algorithm as its replication policy.
2. For supporting high availability via standby engine (journal client), the proposal is electing a leader for all engines.

If so, I suggest that we separate these two focuses and resolve them one after the other.

1. QuorumBasedJournal: a client side implementation that dealing with persisting data into journal server cluster.
2. ...
   • LunaEngine (or its internal JournalWriter, concretely) has a leader election mechanism for determining which instance is the exclusive writer of current shard, with a monotonically increasing token. When a new leader elected, it writes its token to the journal server cluster (quorum).
   • FencedJournal: a server side journal implementation that accepted monotonically increasing token. It only serves requests configured with the current token and accepts monotonically increasing only token updates.

[huachaohuang]

This seems a bit over design, at least for use cases similar to the luna engine. The semantic of LeaderBasedJournal is easier to use and the previous design in this RFC naturally maps to the API. It is OK to decouple the internal implementation to some reusable components like QuorumBasedJournal + FencedJournal, but that depends on the implementation.

[w41ter]

I have divided this RPC into two parts. This part mainly focuses on the design of the leader based journal, the design of the shared journal will be submitted with a new RPC later.

cc @huachaohuang @tisonkun @zojw

[huachaohuang]

This file is unused in this document.

[huachaohuang]

I think we need some API like wait_next here so that the caller can wait for the next event?

[tisonkun]

As long as it's a stream of state/event, an extension like next() can achieve this out-of-the-box.

@w41ter-l Is "state" just "event" or something different?

[w41ter]

@tisonkun I've added some details, maybe able to answer your question.

[w41ter]

@huachaohuang It seems observe_state is enough to wait for the next event, are there details that I haven't considered?

[huachaohuang]

Maybe watch_state is better?

[w41ter]

There doesn't seem to be a difference between the two, is there something I haven't noticed?

[tisonkun]

LGTM. Nits inline.

[tisonkun]

You can discuss the semantic of APIs in sections instead of paragraphs, which separates focuses better.

Also, for significant procedure, such as read/append events, leader state changes, and recovery, you'd better include pseudo code of the procedure or a diagram. For example, in #287 I can see a new concept Phase that participates the recovery and leader state changes. I guess it's part of significants of this RFC?

[w41ter]

Thinks for you advices, I have separated paragraphs into sections.

I think the new concept is the implementation details, so I will introduce it in a follow up RFC.

[tisonkun]

According to the following API, role binds to a stream of the journal. Do you intend to elect leader per stream (a.k.a., in stream granularity)?

Here you write leader and followers bind to a journal, though.

[w41ter]

Thinks, I will fix it.

[zojw]

how do we handle connection reconnection after a stream is returned?

is the caller's duty to retry or retry logic is done inside the returned stream?

[w41ter]

retry logic should be done in inside.

[zojw]

So, is the engines called observe_state("stream1") have a chance to be a follower or leader of stream1~? so for another engine that does not interest those stream1 can be avoid be elect as a follower

[w41ter]

Yes, only called observe_state("stream") would be a follower or leader of stream.

[zojw]

what's the content of String, is it need to be logic server-id or something else~?

[w41ter]

I have no idea about which is better to fill, so I just left it a String. it can be changed once we found and defines a properly structs.

[huachaohuang]

Closed in favor of #361. Please help to improve the stream store design in #361.