Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Example RedisCluster is not working #1168

Open
sebt3 opened this issue Dec 14, 2024 · 0 comments
Open

Example RedisCluster is not working #1168

sebt3 opened this issue Dec 14, 2024 · 0 comments
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@sebt3
Copy link

sebt3 commented Dec 14, 2024

What version of redis operator are you using?

kubectl logs -n kydah-dbo redis-operator-854bf96984-v6d8b |grep rennes-dev
{"level":"info","ts":1734178862.9923024,"logger":"controllers.RedisCluster","msg":"Reconciling opstree redis Cluster controller","Request.Namespace":"rennes-dev","Request.Name":"redis-cluster"}
{"level":"info","ts":1734178863.0176885,"logger":"controller_redis","msg":"Redis statefulset get action failed","Request.StatefulSet.Namespace":"rennes-dev","Request.StatefulSet.Name":"redis-cluster-leader"}
{"level":"info","ts":1734178863.032022,"logger":"controller_redis","msg":"Redis service get action is failed","Request.Service.Namespace":"rennes-dev","Request.Service.Name":"redis-cluster-leader-headless"}
{"level":"info","ts":1734178863.046514,"logger":"controller_redis","msg":"Redis service get action is failed","Request.Service.Namespace":"rennes-dev","Request.Service.Name":"redis-cluster-leader"}
{"level":"info","ts":1734178863.0701113,"logger":"controller_redis","msg":"Redis service get action is failed","Request.Service.Namespace":"rennes-dev","Request.Service.Name":"redis-cluster-leader-additional"}
{"level":"info","ts":1734178863.086634,"logger":"controller_redis","msg":"Redis PodDisruptionBudget get action failed","Request.PodDisruptionBudget.Namespace":"rennes-dev","Request.PodDisruptionBudget.Name":"redis-cluster-leader"}
{"level":"info","ts":1734178863.0914545,"logger":"controller_redis","msg":"Redis statefulset get action failed","Request.StatefulSet.Namespace":"rennes-dev","Request.StatefulSet.Name":"redis-cluster-follower"}
{"level":"error","ts":1734178863.091487,"logger":"controller.rediscluster","msg":"Reconciler error","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","name":"redis-cluster","namespace":"rennes-dev","error":"statefulsets.apps \"redis-cluster-follower\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1734178863.0915482,"logger":"controllers.RedisCluster","msg":"Reconciling opstree redis Cluster controller","Request.Namespace":"rennes-dev","Request.Name":"redis-cluster"}
{"level":"info","ts":1734178863.122763,"logger":"controller_redis","msg":"Redis PodDisruptionBudget get action failed","Request.PodDisruptionBudget.Namespace":"rennes-dev","Request.PodDisruptionBudget.Name":"redis-cluster-leader"}
{"level":"info","ts":1734178863.1283746,"logger":"controller_redis","msg":"Redis statefulset get action failed","Request.StatefulSet.Namespace":"rennes-dev","Request.StatefulSet.Name":"redis-cluster-follower"}
{"level":"error","ts":1734178863.1284044,"logger":"controller.rediscluster","msg":"Reconciler error","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","name":"redis-cluster","namespace":"rennes-dev","error":"statefulsets.apps \"redis-cluster-follower\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1734178863.1284575,"logger":"controllers.RedisCluster","msg":"Reconciling opstree redis Cluster controller","Request.Namespace":"rennes-dev","Request.Name":"redis-cluster"}
{"level":"info","ts":1734178863.1611688,"logger":"controller_redis","msg":"Redis PodDisruptionBudget get action failed","Request.PodDisruptionBudget.Namespace":"rennes-dev","Request.PodDisruptionBudget.Name":"redis-cluster-leader"}
{"level":"info","ts":1734178863.165395,"logger":"controller_redis","msg":"Redis statefulset get action failed","Request.StatefulSet.Namespace":"rennes-dev","Request.StatefulSet.Name":"redis-cluster-follower"}
{"level":"error","ts":1734178863.1654243,"logger":"controller.rediscluster","msg":"Reconciler error","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","name":"redis-cluster","namespace":"rennes-dev","error":"statefulsets.apps \"redis-cluster-follower\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1734178863.1654704,"logger":"controllers.RedisCluster","msg":"Reconciling opstree redis Cluster controller","Request.Namespace":"rennes-dev","Request.Name":"redis-cluster"}
{"level":"info","ts":1734178863.2176614,"logger":"controller_redis","msg":"Redis PodDisruptionBudget get action failed","Request.PodDisruptionBudget.Namespace":"rennes-dev","Request.PodDisruptionBudget.Name":"redis-cluster-leader"}
{"level":"info","ts":1734178863.22232,"logger":"controller_redis","msg":"Redis statefulset get action failed","Request.StatefulSet.Namespace":"rennes-dev","Request.StatefulSet.Name":"redis-cluster-follower"}
{"level":"error","ts":1734178863.222351,"logger":"controller.rediscluster","msg":"Reconciler error","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","name":"redis-cluster","namespace":"rennes-dev","error":"statefulsets.apps \"redis-cluster-follower\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1734178863.222404,"logger":"controllers.RedisCluster","msg":"Reconciling opstree redis Cluster controller","Request.Namespace":"rennes-dev","Request.Name":"redis-cluster"}
{"level":"info","ts":1734178863.3127275,"logger":"controller_redis","msg":"Redis PodDisruptionBudget get action failed","Request.PodDisruptionBudget.Namespace":"rennes-dev","Request.PodDisruptionBudget.Name":"redis-cluster-leader"}
{"level":"info","ts":1734178863.3219056,"logger":"controller_redis","msg":"Redis statefulset get action failed","Request.StatefulSet.Namespace":"rennes-dev","Request.StatefulSet.Name":"redis-cluster-follower"}
{"level":"error","ts":1734178863.3219402,"logger":"controller.rediscluster","msg":"Reconciler error","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","name":"redis-cluster","namespace":"rennes-dev","error":"statefulsets.apps \"redis-cluster-follower\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1734178863.3220003,"logger":"controllers.RedisCluster","msg":"Reconciling opstree redis Cluster controller","Request.Namespace":"rennes-dev","Request.Name":"redis-cluster"}
{"level":"info","ts":1734178863.3975441,"logger":"controller_redis","msg":"Redis PodDisruptionBudget get action failed","Request.PodDisruptionBudget.Namespace":"rennes-dev","Request.PodDisruptionBudget.Name":"redis-cluster-leader"}
{"level":"info","ts":1734178863.405034,"logger":"controller_redis","msg":"Redis statefulset get action failed","Request.StatefulSet.Namespace":"rennes-dev","Request.StatefulSet.Name":"redis-cluster-follower"}
{"level":"error","ts":1734178863.4050715,"logger":"controller.rediscluster","msg":"Reconciler error","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","name":"redis-cluster","namespace":"rennes-dev","error":"statefulsets.apps \"redis-cluster-follower\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1734178863.4051323,"logger":"controllers.RedisCluster","msg":"Reconciling opstree redis Cluster controller","Request.Namespace":"rennes-dev","Request.Name":"redis-cluster"}
{"level":"info","ts":1734178863.440944,"logger":"controller_redis","msg":"Redis PodDisruptionBudget get action failed","Request.PodDisruptionBudget.Namespace":"rennes-dev","Request.PodDisruptionBudget.Name":"redis-cluster-leader"}
{"level":"info","ts":1734178863.4475963,"logger":"controller_redis","msg":"Redis statefulset get action failed","Request.StatefulSet.Namespace":"rennes-dev","Request.StatefulSet.Name":"redis-cluster-follower"}
{"level":"error","ts":1734178863.447642,"logger":"controller.rediscluster","msg":"Reconciler error","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","name":"redis-cluster","namespace":"rennes-dev","error":"statefulsets.apps \"redis-cluster-follower\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1734178863.5653358,"logger":"controllers.RedisCluster","msg":"Reconciling opstree redis Cluster controller","Request.Namespace":"rennes-dev","Request.Name":"redis-cluster"}
{"level":"info","ts":1734178863.6085036,"logger":"controller_redis","msg":"Redis PodDisruptionBudget get action failed","Request.PodDisruptionBudget.Namespace":"rennes-dev","Request.PodDisruptionBudget.Name":"redis-cluster-leader"}
{"level":"info","ts":1734178863.6148853,"logger":"controller_redis","msg":"Redis statefulset get action failed","Request.StatefulSet.Namespace":"rennes-dev","Request.StatefulSet.Name":"redis-cluster-follower"}
{"level":"error","ts":1734178863.6149182,"logger":"controller.rediscluster","msg":"Reconciler error","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","name":"redis-cluster","namespace":"rennes-dev","error":"statefulsets.apps \"redis-cluster-follower\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1734178864.2550938,"logger":"controllers.RedisCluster","msg":"Reconciling opstree redis Cluster controller","Request.Namespace":"rennes-dev","Request.Name":"redis-cluster"}
{"level":"info","ts":1734178864.298574,"logger":"controller_redis","msg":"Redis PodDisruptionBudget get action failed","Request.PodDisruptionBudget.Namespace":"rennes-dev","Request.PodDisruptionBudget.Name":"redis-cluster-leader"}
{"level":"info","ts":1734178864.305149,"logger":"controller_redis","msg":"Redis statefulset get action failed","Request.StatefulSet.Namespace":"rennes-dev","Request.StatefulSet.Name":"redis-cluster-follower"}
{"level":"error","ts":1734178864.3051844,"logger":"controller.rediscluster","msg":"Reconciler error","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","name":"redis-cluster","namespace":"rennes-dev","error":"statefulsets.apps \"redis-cluster-follower\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1734178865.5853336,"logger":"controllers.RedisCluster","msg":"Reconciling opstree redis Cluster controller","Request.Namespace":"rennes-dev","Request.Name":"redis-cluster"}
{"level":"info","ts":1734178865.6274717,"logger":"controller_redis","msg":"Redis PodDisruptionBudget get action failed","Request.PodDisruptionBudget.Namespace":"rennes-dev","Request.PodDisruptionBudget.Name":"redis-cluster-leader"}
{"level":"info","ts":1734178865.635912,"logger":"controller_redis","msg":"Redis statefulset get action failed","Request.StatefulSet.Namespace":"rennes-dev","Request.StatefulSet.Name":"redis-cluster-follower"}
{"level":"error","ts":1734178865.6359558,"logger":"controller.rediscluster","msg":"Reconciler error","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","name":"redis-cluster","namespace":"rennes-dev","error":"statefulsets.apps \"redis-cluster-follower\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1734178868.1964955,"logger":"controllers.RedisCluster","msg":"Reconciling opstree redis Cluster controller","Request.Namespace":"rennes-dev","Request.Name":"redis-cluster"}
{"level":"info","ts":1734178868.2360623,"logger":"controller_redis","msg":"Redis PodDisruptionBudget get action failed","Request.PodDisruptionBudget.Namespace":"rennes-dev","Request.PodDisruptionBudget.Name":"redis-cluster-leader"}
{"level":"info","ts":1734178868.2423453,"logger":"controller_redis","msg":"Redis statefulset get action failed","Request.StatefulSet.Namespace":"rennes-dev","Request.StatefulSet.Name":"redis-cluster-follower"}
{"level":"error","ts":1734178868.2423747,"logger":"controller.rediscluster","msg":"Reconciler error","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","name":"redis-cluster","namespace":"rennes-dev","error":"statefulsets.apps \"redis-cluster-follower\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1734178873.3634644,"logger":"controllers.RedisCluster","msg":"Reconciling opstree redis Cluster controller","Request.Namespace":"rennes-dev","Request.Name":"redis-cluster"}
{"level":"info","ts":1734178873.4082146,"logger":"controller_redis","msg":"Redis PodDisruptionBudget get action failed","Request.PodDisruptionBudget.Namespace":"rennes-dev","Request.PodDisruptionBudget.Name":"redis-cluster-leader"}
{"level":"info","ts":1734178873.4146829,"logger":"controller_redis","msg":"Redis statefulset get action failed","Request.StatefulSet.Namespace":"rennes-dev","Request.StatefulSet.Name":"redis-cluster-follower"}
{"level":"error","ts":1734178873.4147158,"logger":"controller.rediscluster","msg":"Reconciler error","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","name":"redis-cluster","namespace":"rennes-dev","error":"statefulsets.apps \"redis-cluster-follower\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}

redis-operator version: quay.io/opstree/redis-operator:v0.15.1

Does this issue reproduce with the latest release?
As far as I know ;)

What operating system and processor architecture are you using (kubectl version)?

kubectl version Output
$ kubectl version
Client Version: v1.30.6
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.6

What did you do?

kubectl create RedisCluster command
$ kubectl apply -f - <<ENDF
apiVersion: redis.redis.opstreelabs.in/v1beta2
kind: RedisCluster
metadata:
  name: redis-cluster
#  annotations:
#    rediscluster.opstreelabs.in/skip-reconcile: "true"
spec:
  clusterSize: 3
  clusterVersion: v7
  podSecurityContext:
    runAsUser: 1000
    fsGroup: 1000
  persistenceEnabled: true
  kubernetesConfig:
    image: quay.io/opstree/redis:v7.0.12
    imagePullPolicy: IfNotPresent
  redisExporter:
    enabled: false
    image: quay.io/opstree/redis-exporter:v1.44.0
    imagePullPolicy: IfNotPresent
  storage:
    volumeClaimTemplate:
      spec:
        # storageClassName: standard
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi
    nodeConfVolumeClaimTemplate:
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi
            # storageClassName: standard
ENDF

What did you expect to see?
A working cluster. This is from the examples...

What did you see instead?

kubectl get po command
$ kubectl get po
NAME                     READY   STATUS             RESTARTS       AGE
redis-cluster-leader-0   0/1     CrashLoopBackOff   5 (105s ago)   4m52s
kubectl logs redis-cluster-leader-0 command
$ kubectl logs redis-cluster-leader-0 
mkdir: can't create directory '/node-conf': Permission denied
Redis is running without password which is not recommended
sed: /node-conf/nodes.conf: No such file or directory
Running without TLS mode
ACL_MODE is not true, skipping ACL file modification
Starting redis service in cluster mode.....
7:C 14 Dec 2024 12:27:01.553 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
7:C 14 Dec 2024 12:27:01.553 # Redis version=7.0.12, bits=64, commit=00000000, modified=0, pid=7, just started
7:C 14 Dec 2024 12:27:01.553 # Configuration loaded
7:M 14 Dec 2024 12:27:01.553 * monotonic clock: POSIX clock_gettime
7:M 14 Dec 2024 12:27:01.553 # Can't open /node-conf/nodes.conf in order to acquire a lock: No such file or directory

The RedisCluster is created using the recommanded podSecurityContext yet before even starting we can see that

mkdir: can't create directory '/node-conf': Permission denied

So the cluster will go nowhere. what am I doing wrong ?
Using an other storageClass doesnt help

@sebt3 sebt3 added the bug Something isn't working label Dec 14, 2024
@drivebyer drivebyer added the help wanted Extra attention is needed label Dec 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants