Skip to content
This repository has been archived by the owner on Sep 2, 2022. It is now read-only.

Flink Operator installation is failing #467

Open
sumchak1 opened this issue Jul 30, 2021 · 1 comment
Open

Flink Operator installation is failing #467

sumchak1 opened this issue Jul 30, 2021 · 1 comment

Comments

@sumchak1
Copy link

Hi Team, Flink operator installation in IBM cloud is failing with CrashLoopBackOff error. Please see below for more details:

$ k get all -n flink-operator-system
NAME                                                     READY   STATUS             RESTARTS   AGE
pod/cert-job-wtvwr                                       0/1     Completed          0          11m
pod/flink-operator-controller-manager-848b69b444-86bf2   1/2     CrashLoopBackOff   5          4m29s

NAME                                                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/flink-operator-controller-manager-metrics-service   ClusterIP   172.21.152.91    <none>        8443/TCP   26m
service/flink-operator-webhook-service                      ClusterIP   172.21.210.155   <none>        443/TCP    26m

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/flink-operator-controller-manager   0/1     1            0           26m

NAME                                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/flink-operator-controller-manager-848b69b444   1         1         0       26m

NAME                 COMPLETIONS   DURATION   AGE
job.batch/cert-job   1/1           3s         11m
$ k logs flink-operator-controller-manager-848b69b444-t5k2n -n flink-operator-system
Error from server (NotFound): pods "flink-operator-controller-manager-848b69b444-t5k2n" not found
[sumit@sumit flink]$ k logs flink-operator-controller-manager-848b69b444-86bf2 -n flink-operator-system --all-containers
I0729 12:43:04.599359       1 main.go:209] Generating self signed cert as no cert is provided
I0729 12:43:04.820946       1 main.go:242] Listening securely on 0.0.0.0:8443
W0729 12:49:19.067631       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2021-07-29T12:49:20.074Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": "127.0.0.1:8080"}
2021-07-29T12:49:20.074Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "flinkoperator.k8s.io/v1beta1, Kind=FlinkCluster", "path": "/mutate-flinkoperator-k8s-io-v1beta1-flinkcluster"}
2021-07-29T12:49:20.074Z	INFO	controller-runtime.webhook	registering webhook	{"path": "/mutate-flinkoperator-k8s-io-v1beta1-flinkcluster"}
2021-07-29T12:49:20.074Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "flinkoperator.k8s.io/v1beta1, Kind=FlinkCluster", "path": "/validate-flinkoperator-k8s-io-v1beta1-flinkcluster"}
2021-07-29T12:49:20.074Z	INFO	controller-runtime.webhook	registering webhook	{"path": "/validate-flinkoperator-k8s-io-v1beta1-flinkcluster"}
2021-07-29T12:49:20.074Z	INFO	setup	Starting manager
2021-07-29T12:49:20.106Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
2021-07-29T12:49:20.106Z	INFO	controller-runtime.controller	Starting EventSource	{"controller": "flinkcluster", "source": "kind source: /, Kind="}
2021-07-29T12:49:20.106Z	INFO	controller-runtime.webhook.webhooks	starting webhook server
2021-07-29T12:49:20.127Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
2021-07-29T12:49:20.127Z	INFO	controller-runtime.webhook	serving webhook server	{"host": "", "port": 443}
2021-07-29T12:49:20.135Z	INFO	controller-runtime.certwatcher	Starting certificate watcher
2021-07-29T12:49:20.307Z	INFO	controller-runtime.controller	Starting EventSource	{"controller": "flinkcluster", "source": "kind source: /, Kind="}
2021-07-29T12:49:21.208Z	INFO	controller-runtime.controller	Starting EventSource	{"controller": "flinkcluster", "source": "kind source: /, Kind="}
2021-07-29T12:49:21.309Z	INFO	controller-runtime.controller	Starting EventSource	{"controller": "flinkcluster", "source": "kind source: /, Kind="}
 
$ k describe pod flink-operator-controller-manager-848b69b444-86bf2 -n flink-operator-system
Name:               flink-operator-controller-manager-848b69b444-86bf2
Namespace:          flink-operator-system
Priority:           0
PriorityClassName:  <none>
Node:               10.148.145.115/10.148.145.115
Start Time:         Thu, 29 Jul 2021 18:13:02 +0530
Labels:             app=flink-operator
                    control-plane=controller-manager
                    pod-template-hash=848b69b444
Annotations:        cni.projectcalico.org/podIP: 172.30.149.218/32
                    cni.projectcalico.org/podIPs: 172.30.149.218/32
                    kubernetes.io/psp: ibm-privileged-psp
Status:             Running
IP:                 172.30.149.218
Controlled By:      ReplicaSet/flink-operator-controller-manager-848b69b444
Containers:
  kube-rbac-proxy:
    Container ID:  containerd://85ff7f6cf568dc376a0b248be8022e81bd3da48c83ea4461f050694c4a22acec
    Image:         gcr.io/kubebuilder/kube-rbac-proxy:v0.4.0
    Image ID:      gcr.io/kubebuilder/kube-rbac-proxy@sha256:297896d96b827bbcb1abd696da1b2d81cab88359ac34cce0e8281f266b4e08de
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      --secure-listen-address=0.0.0.0:8443
      --upstream=http://127.0.0.1:8080/
      --logtostderr=true
      --v=10
    State:          Running
      Started:      Thu, 29 Jul 2021 18:13:04 +0530
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-5c4cn (ro)
  flink-operator:
    Container ID:  containerd://2880f9d0de68228476738c8ce01d6f82c36149709e60eb33b014b1aaad19a073
    Image:         gcr.io/flink-operator/flink-operator:latest
    Image ID:      gcr.io/flink-operator/flink-operator@sha256:af78aef1e6ca3e082f5d03b53db09fe0d31e21424ac87c9f0204b3739001d3cc
    Port:          443/TCP
    Host Port:     0/TCP
    Command:
      /flink-operator
    Args:
      --metrics-addr=127.0.0.1:8080
      --watch-namespace=
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Thu, 29 Jul 2021 18:16:32 +0530
      Finished:     Thu, 29 Jul 2021 18:16:36 +0530
    Ready:          False
    Restart Count:  5
    Limits:
      cpu:     100m
      memory:  30Mi
    Requests:
      cpu:        100m
      memory:     20Mi
    Environment:  <none>
    Mounts:
      /tmp/k8s-webhook-server/serving-certs from cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-5c4cn (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  webhook-server-cert
    Optional:    false
  default-token-5c4cn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-5c4cn
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 600s
                 node.kubernetes.io/unreachable:NoExecute for 600s
Events:
  Type     Reason     Age                    From                     Message
  ----     ------     ----                   ----                     -------
  Normal   Scheduled  4m8s                   default-scheduler        Successfully assigned flink-operator-system/flink-operator-controller-manager-848b69b444-86bf2 to 10.148.145.115
  Normal   Pulled     4m6s                   kubelet, 10.148.145.115  Container image "gcr.io/kubebuilder/kube-rbac-proxy:v0.4.0" already present on machine
  Normal   Created    4m6s                   kubelet, 10.148.145.115  Created container kube-rbac-proxy
  Normal   Started    4m6s                   kubelet, 10.148.145.115  Started container kube-rbac-proxy
  Normal   Pulled     4m6s                   kubelet, 10.148.145.115  Successfully pulled image "gcr.io/flink-operator/flink-operator:latest" in 233.095305ms
  Normal   Pulled     4m                     kubelet, 10.148.145.115  Successfully pulled image "gcr.io/flink-operator/flink-operator:latest" in 326.358312ms
  Normal   Pulled     3m42s                  kubelet, 10.148.145.115  Successfully pulled image "gcr.io/flink-operator/flink-operator:latest" in 241.20523ms
  Normal   Created    3m9s (x4 over 4m6s)    kubelet, 10.148.145.115  Created container flink-operator
  Normal   Started    3m9s (x4 over 4m5s)    kubelet, 10.148.145.115  Started container flink-operator
  Normal   Pulling    3m9s (x4 over 4m6s)    kubelet, 10.148.145.115  Pulling image "gcr.io/flink-operator/flink-operator:latest"
  Normal   Pulled     3m9s                   kubelet, 10.148.145.115  Successfully pulled image "gcr.io/flink-operator/flink-operator:latest" in 230.947675ms
  Warning  BackOff    2m40s (x6 over 3m53s)  kubelet, 10.148.145.115  Back-off restarting failed container
@fallen-up
Copy link

fallen-up commented Dec 16, 2021 •
edited
Loading

@sumchak1 see, your describe output:

 State:          Waiting
   Reason:       CrashLoopBackOff
 Last State:     Terminated
   Reason:       OOMKilled

Try change limits&requests:
https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/blob/master/helm-chart/flink-operator/templates/flink-operator.yaml#L346-L351

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@fallen-up @sumchak1