For example, a simple readiness probe to ensure pods only roll when it is safe to do so:

#!/bin/sh

state=$(redis-cli cluster info | grep cluster_state | cut -d: -f2 | tr -d '[:space:]')

if [ "$state" != ok ]; then
  echo "FAIL: Cluster state is $state"
  exit 1
fi

slots_assigned=$(redis-cli cluster info | grep cluster_slots_assigned | cut -d: -f2 | tr -d '[:space:]')
slots_ok=$(redis-cli cluster info | grep cluster_slots_ok | cut -d: -f2 | tr -d '[:space:]')

if [ "$slots_assigned" != "$slots_ok" ]; then
  echo "FAIL: Not all assigned cluster slots are ok"
  exit 1
fi

But this will fail until the cluster is actually created and it will fail on nodes being added/removed from the cluster, making it so the operator halts any progress.

Currently, we utilize this state from

Unfortunately without a readiness probe, the StatefulSet has no protections against rolling the pods too fast, so we also need readiness probes (unless the operator can somehow add a feature to control how fast the pods roll in a RollingUpdate)

I think #923 is somewhat related, although that talks about solving this in the operator itself versus just properly allowing something like a readiness probe to be used

There is also another issue that occurs when you don't use readiness probes, scaling up the cluster causes requests to be sent to the new leader/follower pods even before they have joined the cluster, causing request errors.

@drivebyer Still working on actually testing it, but curious what you think of #1038 as a potential way to fix this

@drivebyer Still working on actually testing it, but curious what you think of #1038 as a potential way to fix this

I will look into this by this weekend.