Raft consensus for lotus nodes in a cluster #9294

shrenujbansal · 2022-09-12T20:04:01Z

Related Issues

Proposed Changes

Adding raft consensus to lotus nodes to maintain consistent state for nonces and messages being published to the lotus node

Additional Info

Checklist

Before you mark the PR ready for review, please make sure that:

All commits have a clear commit message.
The PR title is in the form of of <PR type>: <area>: <change being made>
- example: fix: mempool: Introduce a cache for valid signatures
- PR type: fix, feat, INTERFACE BREAKING CHANGE, CONSENSUS BREAKING, build, chore, ci, docs,perf, refactor, revert, style, test
- area: api, chain, state, vm, data transfer, market, mempool, message, block production, multisig, networking, paychan, proving, sealing, wallet, deps
This PR has tests for new functionality or change in behaviour
If new user-facing features are introduced, clear usage guidelines and / or documentation updates should be included in https://lotus.filecoin.io or Discussion Tutorials.
CI is green

…or-chain-nodes

itests/kit/node_full.go

…or-chain-nodes

magik6k

First pass, doesn't feel like there's that much left to do, mostly cleaning up / making stuff more robust.

Looks like a devnet genesis (devgen.car) got committed by accident; Given that it's a little bit we'll need to make sure it's removed from all commits in the PR to not carry it with git history after merging

localnet.json

chain/messagesigner/messagesigner.go

node/config/types.go

node/modules/rpc.go

…or-chain-nodes

chain/messagesigner/messagesigner.go

magik6k · 2022-10-14T15:25:51Z

chain/messagesigner/messagesigner_consensus.go

+func (ms *MessageSignerConsensus) RedirectToLeader(ctx context.Context, method string, arg interface{}, ret interface{}) (bool, error) {
+	ok, err := ms.consensus.RedirectToLeader(method, arg, ret.(*types.SignedMessage))
+	if err != nil {
+		return ok, err
+	}
+	return ok, nil
+}


Can we drop this method?

Suggested change

func (ms *MessageSignerConsensus) RedirectToLeader(ctx context.Context, method string, arg interface{}, ret interface{}) (bool, error) {

ok, err := ms.consensus.RedirectToLeader(method, arg, ret.(*types.SignedMessage))

if err != nil {

return ok, err

}

return ok, nil

}

No, this is needed to be called in MpoolPushMessage

chain/messagesigner/messagesigner_consensus.go

cli/util/api.go

cli/util/apiinfo.go

documentation/en/default-lotus-config.toml

node/modules/storageminer.go

…or-chain-nodes

magik6k

Looks like there is a commit updating extern/filecoin-ffi by accident

magik6k · 2022-11-14T15:16:25Z

api/types.go

+type NonceMapType map[address.Address]uint64
+type MsgUuidMapType map[uuid.UUID]*types.SignedMessage


Not needed to land this PR, but if we wanted to be fancy, we could write a generic helper type for maps with typed keys.

chain/messagepool/provider.go

chain/messagesigner/messagesigner.go

magik6k · 2022-11-14T15:26:51Z

chain/messagesigner/messagesigner_consensus.go

+	ds dtypes.MetadataDS,
+	consensus *consensus.Consensus) *MessageSignerConsensus {
+
+	ds = namespace.Wrap(ds, datastore.NewKey("/message-signer-consensus/"))


This is a different prefix than the one used in non-consensus messagesigner, but I guess that's what we want.

I'm just not 100% sure that we actually need this to be a different prefix.

documentation/en/default-lotus-config.toml

lib/consensus/raft/consensus.go

lib/consensus/raft/raft.go

…or-chain-nodes

geoff-vball · 2022-11-15T21:43:23Z

chain/messagepool/messagepool.go

@@ -667,7 +668,7 @@ func (mp *MessagePool) verifyMsgBeforeAdd(ctx context.Context, m *types.SignedMe
 	return publish, nil
 }

-func (mp *MessagePool) Push(ctx context.Context, m *types.SignedMessage) (cid.Cid, error) {
+func (mp *MessagePool) Push(ctx context.Context, m *types.SignedMessage, publish bool) (cid.Cid, error) {


I don't see a similar change for PushUntrusted. Is that intentional?

Also, could you leave a clear comment on what exactly this bool does?

We shouldn't need it for PushUntrusted since its not used in syncing the message pool. I think this API should really be folded back into Push itself and add a untrusted param to it

Yes I definitely agree :)

magik6k · 2022-11-15T22:12:11Z

Dependency diff if anyone else wants to review: https://gist.github.com/magik6k/ada79bebc4f5ccf3078d72f6513a49e2

shrenujbansal requested a review from a team as a code owner September 12, 2022 20:04

shrenujbansal force-pushed the sbansal/nonce-coordination-and-consensus-for-chain-nodes branch from 1dc9115 to 58013e8 Compare September 12, 2022 20:08

WIP: Raft consensus for lotus nodes in a cluster

8f1b1bb

shrenujbansal force-pushed the sbansal/nonce-coordination-and-consensus-for-chain-nodes branch from 58013e8 to 8f1b1bb Compare September 12, 2022 20:10

Shrenuj Bansal added 2 commits September 12, 2022 16:23

Merge branch 'master' into sbansal/nonce-coordination-and-consensus-f…

a1f2fdb

…or-chain-nodes

Few more changes

4171be0

shrenujbansal commented Sep 12, 2022

View reviewed changes

itests/kit/node_full.go Outdated Show resolved Hide resolved

Shrenuj Bansal added 3 commits September 13, 2022 17:30

WIP: rest of the stuff

3441224

Merge branch 'master' into sbansal/nonce-coordination-and-consensus-f…

4be8861

…or-chain-nodes

Cluster raft config changes

81c729e

magik6k marked this pull request as draft September 16, 2022 10:51

magik6k reviewed Sep 16, 2022

View reviewed changes

Shrenuj Bansal added 2 commits September 21, 2022 15:41

Add Auth func for gorpc and address comments

1fe4aa3

Address moar comments

7470549

magik6k mentioned this pull request Sep 27, 2022

Backport: #9061 rpc errors #9384

Merged

5 tasks

Shrenuj Bansal added 5 commits September 27, 2022 16:08

More wip

99e7c32

Merge branch 'master' into sbansal/nonce-coordination-and-consensus-f…

559c2c6

…or-chain-nodes

Retries within proxy working

570f614

Add Mpool ref to raft state and rearrange some APIs

f89a682

Add persistent stores for cluster raft data

b8060cd

Fatman13 mentioned this pull request Sep 30, 2022

[RPC] RPC支持加入错误类型error type？ filecoin-project/venus#5334

Closed

1 task

Shrenuj Bansal added 4 commits September 30, 2022 16:45

Use multiaddrs in config for raft peerset

986c5e3

Change Mpool push API to have an option to publish

dde204f

solution for mining loop hitting the same node

9848182

Ignore mpool msg existing errors for applying raft state

17a7722

magik6k reviewed Oct 14, 2022

View reviewed changes

Shrenuj Bansal added 4 commits October 16, 2022 22:52

fix some bugs and address some comments

139f877

Change cli cmd to API with proxy

b77ca54

some cleanup

900525f

fix lint and make gen

674427a

Shrenuj Bansal added 3 commits October 18, 2022 15:53

Address more comments and add test for gorpc auth

ad8b959

make gen

94bd4d8

i hate make gen

09e9562

shrenujbansal marked this pull request as ready for review October 20, 2022 14:28

shrenujbansal changed the title ~~WIP: Raft consensus for lotus nodes in a cluster~~ Raft consensus for lotus nodes in a cluster Nov 11, 2022

Shrenuj Bansal added 4 commits November 11, 2022 14:41

Merge branch 'master' into sbansal/nonce-coordination-and-consensus-f…

2fa21ff

…or-chain-nodes

Change config name from Raft to Cluster

2681c2a

remove 2nd rpc closer call

8740fb4

Remove double stop

b541cf9

magik6k reviewed Nov 14, 2022

View reviewed changes

Shrenuj Bansal added 8 commits November 14, 2022 15:46

Address comments

800d9de

update filecoin-ffi

a66619f

one more

ab1eeeb

make gen and docsgen

f14a25a

Merge branch 'master' into sbansal/nonce-coordination-and-consensus-f…

b95d1a6

…or-chain-nodes

remove comments

4b11b45

remove moar commented out code

9451221

Remove some configs

c0925ff

geoff-vball reviewed Nov 15, 2022

View reviewed changes

Add comment to Push API

22f3fbb

geoff-vball approved these changes Nov 15, 2022

View reviewed changes

shrenujbansal merged commit 04f0af5 into master Nov 15, 2022

shrenujbansal deleted the sbansal/nonce-coordination-and-consensus-for-chain-nodes branch November 15, 2022 22:25

Fatman13 mentioned this pull request Nov 21, 2022

[chain-co] lotus多节点方案调研 filecoin-project/venus#5497

Closed

1 task

LexLuthr mentioned this pull request Nov 22, 2022

Allow Boost to use multi-node FULL_NODE_API environment variable filecoin-project/boost#983

Open

magik6k mentioned this pull request Dec 2, 2022

fix: cliutil: Fix URL-based API endpoint parsing #9780

Merged

7 tasks

Fatman13 mentioned this pull request Dec 19, 2022

[docs] chain-co VS. raft comparison / chain-co和raft比较 filecoin-project/venus#5583

Closed

2 tasks

jennijuju mentioned this pull request Mar 29, 2023

Lotus Cluster - consistent view of the chain #10582

Open

magik6k mentioned this pull request Nov 29, 2023

misc: Drop the raft-cluster experiment #11468

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Raft consensus for lotus nodes in a cluster #9294

Raft consensus for lotus nodes in a cluster #9294

shrenujbansal commented Sep 12, 2022

magik6k left a comment

magik6k Oct 14, 2022

shrenujbansal Oct 14, 2022

magik6k left a comment

magik6k Nov 14, 2022

magik6k Nov 14, 2022

geoff-vball Nov 15, 2022

geoff-vball Nov 15, 2022

shrenujbansal Nov 15, 2022

geoff-vball Nov 15, 2022

magik6k commented Nov 15, 2022

		type NonceMapType map[address.Address]uint64
		type MsgUuidMapType map[uuid.UUID]*types.SignedMessage

Raft consensus for lotus nodes in a cluster #9294

Raft consensus for lotus nodes in a cluster #9294

Conversation

shrenujbansal commented Sep 12, 2022

Related Issues

Proposed Changes

Additional Info

Checklist

magik6k left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

magik6k left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

magik6k commented Nov 15, 2022