Lotus Cluster - consistent view of the chain

[jennijuju]

Background

In Nov 2022, we implemented lotus node clusters to maintain a consistent state for nonces and messages being published across a cluster of nodes, and the process may switch between nodes as needed/when a particular node is down.

The lotus cluster was initially designed and implemented for the lotus miner process, which currently has a gap where each process can only talk to one node instead of being able to hit different nodes in the cluster randomly.

Main PRs: #9294, #9174, #9177

The feature is currently in alpha.

Problem

We have a new user group, API service providers, that can leverage from lotus cluster to handle 100k req/sec. Currently, providers are running a cluster of lotus daemon. They are facing classic consistency issues of response when routing requests to different nodes due to inconsistent sync status, reorg, etc. A lotus cluster with a consistent view of the chain would be super helpful for service providers as they could then operate a cluster of lotus clusters (daemon/gateway) + load balancers.

Currently, while the lotus cluster can get the same nonce/messages, but it is not smart enough to provide a consistent view of all filecoin states. We will need to design and implement it.

relevant thread