[Help]: Redis Cluster Disaster Recovery Backup/ restore

[shubham-cmyk]

I have tried simulating backup and restore on the redis cluster.

There seems to be a few limitation on which I am stuck.

This post would contain 2 part taking backup and restoring.

First part is taking backup in redis seems to be a straight forward approach that could be done by first taking a snapshot and then taking a backup of the dump ( snapshot ) to S3 etc.
Referring : https://redis.io/redis-enterprise/technology/backup-disaster-recovery/

Second part would be restoring the dump.
This is the challenging part Since in the redis cluster specifically need to be formed with the empty nodes. If we try to form a cluster with non empty nodes we would get error like :

 [ERR] Node 172.28.0.4:6379 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

In another restore case if we try restoring the cluster configuration along with the dump such that nodes would know each other and their slots.

Every node in the redis cluster would retry to connect to other nodes on the IP's mentioned in the cluster configuration file that would lead to failure since the IP's are different now of every node. These IP's are not static and not managed by operator since we deploy on environment like k8s where the IP's are managed by the CNI and we can't really control the IP's of the pod while in creation.

I would like to hear some suggestion from the community since the disaster recovery procedure in missing in the current operator and there is it an urgent need for this.