Principle of Active and Standby

Architecture

• Pods are distributed on different physical machines, and each Pod contains redis-docker (2 PODs) or sentinel-docker (3 PODs).
• The application only needs to configure redis VIP (ClusterIP) to access the Redis cluster.
• Redis/sentinel executes the deploy program when the container is up to initialize and join the cluster.
• Using StatefulSet of kubernetes, the startup sequence is in order, the Pod starts in an orderly manner, and the Redis container starts after the Sentinel starts.
• Dispatch the scheduling as much as possible, support SSD optional, and IO controllable.

A few notes:

• The previous request routing does not introduce an agent, and only the service judges whether the master info is the master to switch autonomously according to readniess.
• Only the VIP address is configured for the client connection, and the traditional sentinel connection method is not used because the sentinel is fully suspended during the actual online operation and maintenance process and the IP of the active and standby nodes also changes. The client watch mechanism will lose the real master node. The address information cannot be self-healed, and the configuration can only be actively pulled by reapplying.

Implementation process

• The redis container needs to wait for all sentinel container replica sets to start up and register the empty sentinel.
• For sentinel with non-empty registration information, if the registration information is abnormal, redis deploy will clear the sentinel information for new registration when the redis container starts. If it's ok, then deploy will find the master node to join as a slave.
• When encountering a catastrophic situation where the primary and secondary nodes hang at the same time, the first master node to start will pull the remote backup data to load and start.
• When sentinel single point of failure, it will obtain master node information registration from other sentinels, and when all sentinel fails, it will judge whether the master node exists and register.

About scaling

Since Redis configuration items can be dynamically modified, the expansion and contraction (maxmemory) and other configurations can be set through td-redis-operator config set and the configuration file configmap can be changed to be persistently modified.