High Availability Deployment

[sebadob]

Hey guys,

at first, I am really happy, that I found Fluvio.
I am using Nats Jetstream since quite a while now, which is already pretty good.
Since I am doing a lot of IoT / Edge work, it is crucial for me to save as much resources as possible.
Apart from that, I am really passionate about Rust, so it's a win anyway. ;)

I just have a question regarding HA deployment and I did not find something about it in the Docs.
I started with the tutorials and used the fluvio CLI to deploy a cluster, which worked out really nicely, but some things about HA are unclear to me.
I was able to deploy a HA(?) SPU deployment with fluvio cluster start --namespace fluvio --spu-storage-size 1 --spu 3, which is really nice already. But, if I got that right, clients do not communicate with SPUs directly, but though the SC, right? Kind of like the same as TiKV, which has the PD for the communication.
Storagewise, we are good of course with 3 SPUs and a replication factor of 3, but in that case, we only have one SC, which manages the SPUs, and this single SC is the single point of failure, so not really HA.

I did not find anything about it, but is it possible to deploy the SC in some kind of HA mode, be is active / standby or even active / active?
I am currently starting a new really big IoT project with a lot of very small clients in a quite unstable network. Throughput is not the critical factor, but availability is.

Another question would be, if I can configure the failover timeout or something like this, in case the master SPU becomes unavailable. I was connected to a topic in a few consoles and then killed the master SPU pod, which immediately broke the communication (which does make sense with a master / slave model). Only when the master came back online, the communication was working again and the SC did not switch over (probably because it was still too fast?).

The third thing was about authn / authz, was I figured that our with your CI pipeline configs and found out, that it is only working via mTLS and some rolemappings. I guess this will be documented in the future too.

Thanks you for creating this awesome project. I was about to kind of re-implement something like Nats Jetstream in Rust, when I found Fluvio. :)

[ajhunyady]

Hi @sebadob,

Thanks for the message. I'll answer one of the points, and @sehz will answer the others. SC is a proxy for metadata management for SPUs, but the SPUs themselves communicate with the producer/consumers. Checkout this doc for additional details: https://www.fluvio.io/docs/architecture/spu/

[sehz]

Hi @sebadob,

Thanks for your questions! I'm glad to hear that you're finding Fluvio useful and that you're interested in using it for your IoT project.
To answer your questions:

1. Current version of Fluvio doesn't support HA for SC out of the box. However, with small tweak, it should be possible to support SC using K8 Deployment (it already does). We just need to implement a simple health check to ensure SC pod properly fail over. Can you create issue for it?
2. As @ajhunyady indicated, consumer and producer directly to talk to SPU
3. Fluvio implements basic RBAC but requires mTLS. It is implemented in the https://github.com/infinyon/fluvio/tree/master/crates/fluvio-auth. It uses X509 certs to validate security roles. We have implemented simple object permission https://github.com/infinyon/fluvio/blob/master/crates/fluvio-sc/test-data/auth_config/policy.json but could implement more sophisticated permission. We will try to fully describe in the the doc.

Hope this helps! Let me know if you have any other questions.

[sebadob]

Thank you very much for the quick answer.
Yes this helps already.

It's a shame, that I did not see the better description in the architecture overview that @ajhunyady posted, sorry.

From the detailed description I found out, that the SC is always the first contact for newly connecting clients.
Does it mean, that the client behaves in a way, that it at first connects to the SC to fetch metadata about the whole deployment and then actually uses this metadata to connect to the SPUs directly?
I that case, would it be correct, that I need the SC for the initial connection (of course...) and as soon as I have a stable connection, I could actually loose the SC for a few minutes or restart it without any interruption in the Client - SPU communication?
If that is the case, then it would actually be no big deal, if the SC does currently not support a HA mode.

Regarding the authentication - I am planning to create a CA with SSO Login and short lived certificates for TLS and SSH connections in Rust anyway for the next upcoming projects. I guess the client cert, that I provide with the fluvio CLI is just for the CLI itself, right?
If I provide the server cert + key + Server CA + Client CA to the server, I should be able to kind of like renew client certificates every few hours and the server will validate them against the Client Root / Intermediate CA, so it does not need restarts, I guess?

I will have the time to open up an issue about the HA SC after the holidays. Thank you so much already for the fast replies.
I am really looking forward on working with fluvio.

[sehz]

Yes, the client connects to SC to get metadata to talk to SPU. Once clients establish a connection to SPU, disruption to SC doesn't affect the SPU connection.

Certificates are used to establish connections between clients and SC/SPU. Renewing client certs should be ok, as you described, but we haven't thoroughly tested that scenario, so that mileage may vary 🤞.

[sebadob]

That sounds perfect. Thank you very much.
I will start testing soon and then give some feedback.