Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Some pods fail to launch with cgroup related issues #115

Closed
mcastelino opened this issue Jul 15, 2019 · 12 comments
Closed

Some pods fail to launch with cgroup related issues #115

mcastelino opened this issue Jul 15, 2019 · 12 comments
Assignees
@knangia

Comments

@mcastelino
Copy link
Contributor 

Events:
  Type     Reason            Age                    From               Message
  ----     ------            ----                   ----               -------
  Warning  FailedScheduling  2m24s (x2 over 2m24s)  default-scheduler  pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
  Normal   Scheduled         2m23s                  default-scheduler  Successfully assigned monitoring/alertmanager-main-0 to clr-02
  Normal   Pulling           2m12s                  kubelet, clr-02    Pulling image "quay.io/prometheus/alertmanager:v0.15.2"
  Normal   Pulling           117s                   kubelet, clr-02    Pulling image "quay.io/coreos/configmap-reload:v0.0.1"
  Normal   Created           117s                   kubelet, clr-02    Created container alertmanager
  Normal   Started           117s                   kubelet, clr-02    Started container alertmanager
  Normal   Pulled            117s                   kubelet, clr-02    Successfully pulled image "quay.io/prometheus/alertmanager:v0.15.2"
  Normal   Pulled            103s                   kubelet, clr-02    Successfully pulled image "quay.io/coreos/configmap-reload:v0.0.1"
  Warning  Failed            102s                   kubelet, clr-02    Error: container create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/podb7c7c592-fffe-4c98-9d0f-3893b29608d7/crio-cac41a077cb3278857063cff71220123ddb5a0ccec072d69fe35211303ff7525/memory.limit_in_bytes: device or resource busy\\\"\""
  Warning  Failed            102s                   kubelet, clr-02    Error: container create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/podb7c7c592-fffe-4c98-9d0f-3893b29608d7/crio-caf9a842f88d637aeb72da3dc06b7cb22f9b6db3ce59da38b1050f1363f5f78b/memory.limit_in_bytes: device or resource busy\\\"\""
  Warning  Failed            90s                    kubelet, clr-02    Error: container create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/podb7c7c592-fffe-4c98-9d0f-3893b29608d7/crio-c929afacdfe32f13195715bc1380d0c88c2025a0bd2b6ccaa9996ab932ba07a2/memory.limit_in_bytes: device or resource busy\\\"\""
  Warning  Failed            77s                    kubelet, clr-02    Error: container create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/podb7c7c592-fffe-4c98-9d0f-3893b29608d7/crio-e97b2fe36235745d542ab09b030ae3cf76799b4277080208e71424c65ce1b36d/memory.limit_in_bytes: device or resource busy\\\"\""
  Warning  Failed            64s                    kubelet, clr-02    Error: container create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/podb7c7c592-fffe-4c98-9d0f-3893b29608d7/crio-2fe901067800d79944583d8355c1809dba2eccae0524728112cd7e58486a41cb/memory.limit_in_bytes: device or resource busy\\\"\""
  Warning  Failed            52s                    kubelet, clr-02    Error: container create failed: parent closed synchronisation channel
container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/podb7c7c592-fffe-4c98-9d0f-3893b29608d7/crio-2481f7d2cf8f9a5e6a41e97332e98bb10d2650520099683e13ba3e5ef6fa6765/memory.limit_in_bytes: device or resource busy\\\"\""
  Warning  Failed  37s  kubelet, clr-02  Error: container create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/podb7c7c592-fffe-4c98-9d0f-3893b29608d7/crio-eb2d63f5f194f5f38d017a29661f6da270aa65424bfc6f20693ed0d67dcce5d9/memory.limit_in_bytes: device or resource busy\\\"\""
  Warning  Failed  24s  kubelet, clr-02  Error: container create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/podb7c7c592-fffe-4c98-9d0f-3893b29608d7/crio-6ebcc4a386e06e6a6b9b39d583e99270bb82998390a85e78eb8a0c04d78b63a7/memory.limit_in_bytes: device or resource busy\\\"\""
  Warning  Failed  12s  kubelet, clr-02  Error: container create failed: parent closed synchronisation channel
container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/podb7c7c592-fffe-4c98-9d0f-3893b29608d7/crio-bb3d1e3cbb23671ec6b5d044446774d8f7ea3cbffb3a299d7e031191333c85cd/memory.limit_in_bytes: device or resource busy\\\"\""
  Normal   Pulled  0s (x9 over 102s)  kubelet, clr-02  Container image "quay.io/coreos/configmap-reload:v0.0.1" already present on machine
  Warning  Failed  0s                 kubelet, clr-02  (combined from similar events): Error: container create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/podb7c7c592-fffe-4c98-9d0f-3893b29608d7/crio-f21c60bf91c32db9911a41f41e39a82b0255e0a1ad066b173e6896ff7935f85e/memory.limit_in_bytes: device or resource busy\\\"\""
@mcastelino
Copy link
Contributor Author

Another log. This time for prometheus

  Normal   Scheduled         4m56s                  default-scheduler  Successfully assigned monitoring/prometheus-k8s-0 to clr-02
  Normal   Pulling           4m45s                  kubelet, clr-02    Pulling image "quay.io/prometheus/prometheus:v2.4.3"
  Normal   Pulling           4m26s                  kubelet, clr-02    Pulling image "quay.io/coreos/prometheus-config-reloader:v0.25.0"
  Normal   Pulled            4m26s                  kubelet, clr-02    Successfully pulled image "quay.io/prometheus/prometheus:v2.4.3"
  Normal   Created           4m15s                  kubelet, clr-02    Created container prometheus-config-reloader
  Normal   Pulled            4m15s                  kubelet, clr-02    Successfully pulled image "quay.io/coreos/prometheus-config-reloader:v0.25.0"
  Normal   Started           4m15s                  kubelet, clr-02    Started container prometheus-config-reloader
  Warning  Failed            4m14s                  kubelet, clr-02    Error: container create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/pod00ce884e-fb88-4c3a-8a4f-c69e2fe97de8/crio-61b85e24b3b346d688941be096d36b65983b996d4ec5b4f600a5f8de6c584320/memory.limit_in_bytes: device or resource busy\\\"\""
  Normal   Pulled            4m14s                  kubelet, clr-02    Container image "quay.io/prometheus/prometheus:v2.4.3" already present on machine
  Normal   Started           4m13s (x2 over 4m26s)  kubelet, clr-02    Started container prometheus
  Normal   Created           4m13s (x2 over 4m26s)  kubelet, clr-02    Created container prometheus
  Warning  Failed            4m13s                  kubelet, clr-02    Error: container create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/pod00ce884e-fb88-4c3a-8a4f-c69e2fe97de8/crio-1bc776572c74234059c0a000a3cea8e5158fabc15d528423aba1fa9a91b1dbac/memory.limit_in_bytes: device or resource busy\\\"\""
  Warning  Failed            4m12s                  kubelet, clr-02    Error: container create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/pod00ce884e-fb88-4c3a-8a4f-c69e2fe97de8/crio-84be5524d839d245bdb6dc5c4ec62c9ed551002916c7d8a7f811e4611a78d29c/memory.limit_in_bytes: device or resource busy\\\"\""
  Warning  Failed            4m11s                  kubelet, clr-02    Error: container create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/pod00ce884e-fb88-4c3a-8a4f-c69e2fe97de8/crio-5f09b1ebb3f9c39e8e7be3fe453b56c39f3fced561955663ac80779a69f67ff8/memory.limit_in_bytes: device or resource busy\\\"\""
  Warning  Failed            3m58s                  kubelet, clr-02    Error: container create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/pod00ce884e-fb88-4c3a-8a4f-c69e2fe97de8/crio-17cae67649aa68ea1c6afce6aaf1fbb0d6258f31a658397915316402c0359939/memory.limit_in_bytes: device or resource busy\\\"\""
  Warning  Failed            3m44s                  kubelet, clr-02    Error: container create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/pod00ce884e-fb88-4c3a-8a4f-c69e2fe97de8/crio-f43df094f14349a5ebd9c4a9535ced2fd44084d724484a82b864d00466275067/memory.limit_in_bytes: device or resource busy\\\"\""
  Normal   Pulled            3m29s (x7 over 4m15s)  kubelet, clr-02    Container image "quay.io/coreos/configmap-reload:v0.0.1" already present on machine
  Warning  Failed            3m29s                  kubelet, clr-02    Error: container create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/pod00ce884e-fb88-4c3a-8a4f-c69e2fe97de8/crio-6db7e8536a8ceedac313a94b667eb17c3dfdc060e600c16493c24d375ea2e3ae/memory.limit_in_bytes: device or resource busy\\\"\""

@mcastelino
Copy link
Contributor Author

This seems to be a transient condition and eventully as k8s tries to recreate the pod on failure they do come up. It is possible that the containers do not launch due to memory issues which result in the missing cgroup.

Every 2.0s: kubectl get po --all-namespaces -o wide                                                                                                              clr-01: Mon Jul 15 19:18:51 2019

NAMESPACE          NAME                                        READY   STATUS      RESTARTS   AGE     IP                NODE     NOMINATED NODE   READINESS GATES
ingress-nginx      nginx-ingress-controller-765886bf85-2hr5x   1/1     Running     0          12m     10.244.2.15       clr-02   <none>           <none>
ingress-nginx      nginx-ingress-controller-765886bf85-2wzwq   1/1     Running     0          12m     10.244.2.14       clr-02   <none>           <none>
ingress-nginx      nginx-ingress-controller-765886bf85-kxnmw   1/1     Running     0          12m     10.244.1.14       clr-03   <none>           <none>
kube-system        canal-gk8qm                                 3/3     Running     0          31m     192.168.121.189   clr-01   <none>           <none>
kube-system        canal-kvbsl                                 3/3     Running     0          31m     192.168.121.127   clr-03   <none>           <none>
kube-system        canal-zbz7w                                 3/3     Running     0          31m     192.168.121.152   clr-02   <none>           <none>
kube-system        coredns-5c98db65d4-76kd2                    1/1     Running     0          31m     10.244.0.3        clr-01   <none>           <none>
kube-system        coredns-5c98db65d4-gr5wr                    1/1     Running     0          31m     10.244.0.2        clr-01   <none>           <none>
kube-system        elasticsearch-logging-0                     1/1     Running     0          12m     10.244.2.16       clr-02   <none>           <none>
kube-system        elasticsearch-logging-1                     1/1     Running     0          10m     10.244.1.16       clr-03   <none>           <none>
kube-system        etcd-clr-01                                 1/1     Running     0          31m     192.168.121.189   clr-01   <none>           <none>
kube-system        fluentd-es-v2.2.1-57xk4                     1/1     Running     0          12m     10.244.1.12       clr-03   <none>           <none>
kube-system        fluentd-es-v2.2.1-62bs7                     1/1     Running     0          12m     10.244.2.13       clr-02   <none>           <none>
kube-system        kata-deploy-lhk8h                           1/1     Running     0          30m     10.244.1.2        clr-03   <none>           <none>
kube-system        kata-deploy-nd8fr                           1/1     Running     0          30m     10.244.2.2        clr-02   <none>           <none>
kube-system        kibana-logging-df8d4c8fd-6vhhb              1/1     Running     0          12m     10.244.1.13       clr-03   <none>           <none>
kube-system        kube-apiserver-clr-01                       1/1     Running     0          30m     192.168.121.189   clr-01   <none>           <none>
kube-system        kube-controller-manager-clr-01              1/1     Running     0          31m     192.168.121.189   clr-01   <none>           <none>
kube-system        kube-proxy-4d7wg                            1/1     Running     0          31m     192.168.121.152   clr-02   <none>           <none>
kube-system        kube-proxy-jg55g                            1/1     Running     0          31m     192.168.121.189   clr-01   <none>           <none>
kube-system        kube-proxy-mjpcb                            1/1     Running     0          31m     192.168.121.127   clr-03   <none>           <none>
kube-system        kube-scheduler-clr-01                       1/1     Running     0          31m     192.168.121.189   clr-01   <none>           <none>
kube-system        kubernetes-dashboard-74b84cc4b8-scp24       1/1     Running     0          12m     10.244.1.15       clr-03   <none>           <none>
kube-system        metrics-server-96f7df45d-xlwzf              1/1     Running     0          31m     10.244.1.3        clr-03   <none>           <none>
monitoring         alertmanager-main-0                         2/2     Running     0          20m     10.244.2.11       clr-02   <none>           <none>
monitoring         alertmanager-main-1                         2/2     Running     0          16m     10.244.1.11       clr-03   <none>           <none>
monitoring         alertmanager-main-2                         2/2     Running     0          11m     10.244.2.17       clr-02   <none>           <none>
monitoring         grafana-55dcb5484d-vjjrn                    1/1     Running     0          21m     10.244.2.10       clr-02   <none>           <none>
monitoring         kube-state-metrics-55698f6df7-ztkt8         4/4     Running     0          20m     10.244.1.10       clr-03   <none>           <none>
monitoring         node-exporter-2tx7k                         2/2     Running     0          21m     192.168.121.152   clr-02   <none>           <none>
monitoring         node-exporter-g8mgr                         2/2     Running     0          21m     192.168.121.127   clr-03   <none>           <none>
monitoring         node-exporter-kffd8                         2/2     Running     0          21m     192.168.121.189   clr-01   <none>           <none>
monitoring         prometheus-k8s-0                            3/3     Running     1          20m     10.244.2.12       clr-02   <none>           <none>
monitoring         prometheus-k8s-1                            3/3     Running     1          8m19s   10.244.1.17       clr-03   <none>           <none>
monitoring         prometheus-operator-7695b59fb8-jkjtw        1/1     Running     0          21m     10.244.2.9        clr-02   <none>           <none>
rook-ceph-system   rook-ceph-agent-5gh9m                       1/1     Running     0          26m     192.168.121.127   clr-03   <none>           <none>
rook-ceph-system   rook-ceph-agent-6j8w4                       1/1     Running     0          26m     192.168.121.152   clr-02   <none>           <none>
rook-ceph-system   rook-ceph-operator-55f758cfdc-42rmk         1/1     Running     0          27m     10.244.2.3        clr-02   <none>           <none>
rook-ceph-system   rook-discover-c7695                         1/1     Running     0          26m     10.244.2.4        clr-02   <none>           <none>
rook-ceph-system   rook-discover-f4nzt                         1/1     Running     0          26m     10.244.1.4        clr-03   <none>           <none>
rook-ceph          rook-ceph-mgr-a-7bbf985fcc-zmfbx            1/1     Running     0          25m     10.244.1.6        clr-03   <none>           <none>
rook-ceph          rook-ceph-mon0-n2cr2                        1/1     Running     0          26m     10.244.2.5        clr-02   <none>           <none>
rook-ceph          rook-ceph-mon1-9st2t                        1/1     Running     0          25m     10.244.1.5        clr-03   <none>           <none>
rook-ceph          rook-ceph-mon2-sgrnl                        1/1     Running     0          25m     10.244.2.6        clr-02   <none>           <none>
rook-ceph          rook-ceph-osd-id-0-797cfdc9d4-8gjwx         1/1     Running     0          24m     10.244.1.8        clr-03   <none>           <none>
rook-ceph          rook-ceph-osd-id-1-5547668bcb-4jvt8         1/1     Running     0          24m     10.244.2.8        clr-02   <none>           <none>
rook-ceph          rook-ceph-osd-prepare-clr-02-lqzwl          0/1     Completed   0          24m     10.244.2.7        clr-02   <none>           <none>
rook-ceph          rook-ceph-osd-prepare-clr-03-9zlks          0/1     Completed   0          24m     10.244.1.7        clr-03   <none>           <none>

@mcastelino
Copy link
Contributor Author

May be related to opencontainers/runc#1980

@ganeshmaharaj
Copy link
Contributor

This issue seems to happen even with containerd as the runtime. It is also isolated to prometheus-k8s and alertmanager-main pods alone.

@ganeshmaharaj
Copy link
Contributor

The failing pod seems to have a container (possibly spawned by the operator) which has low memory.

 config-reloader:
    Container ID:  containerd://15999b9c17637fbbb2149d93a40e7e24f6fdf1e393a63eef9892825c71d2c2c1
    Image:         quay.io/coreos/configmap-reload:v0.0.1
    Image ID:      sha256:ad5984e09e86bfe2b23e3891066a1e27f4d79dde2f6dbe578b923773ca8462f3
    Port:          <none>
    Host Port:     <none>
    Args:
      -webhook-url=http://localhost:9093/-/reload
      -volume-dir=/etc/alertmanager/config
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       StartError
      Message:      failed to create containerd task: OCI runtime create failed: container_linux.go:348:starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/pod0ee4eabb-3fc5-4014-9fb2-8a5ba5ea121c/15999b9c17637fbbb2149d93a40e7e24f6fdf1e393a63eef9892825c71d2c2c1/memory.limit_in_bytes: device or resource busy\\\"\"": unknown
      Exit Code:    128
      Started:      Thu, 01 Jan 1970 01:00:00 +0100
      Finished:     Thu, 18 Jul 2019 02:44:14 +0100
    Ready:          False
    Restart Count:  5
    Limits:
      cpu:     5m
      memory:  10Mi
    Requests:
      cpu:        5m
      memory:     10Mi
    Environment:  <none>
    Mounts:
      /etc/alertmanager/config from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from alertmanager-main-token-qv5kw (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:

@NitinAtIntel NitinAtIntel self-assigned this Aug 5, 2019
@NitinAtIntel
Copy link
Contributor

I will take a look and verify

@NitinAtIntel NitinAtIntel removed their assignment Sep 16, 2019
@knangia knangia self-assigned this Sep 17, 2019
@knangia
Copy link
Contributor

knangia commented Oct 5, 2019 •
edited
Loading

Not able to reproduce this issue.
All the pods come up and running. Tested with k8s v1.15.3 & v1.16.0 and runc version 1.0.0-rc5.

kubectl get po --all-namespaces -o wide
NAMESPACE     NAME                                            READY   STATUS      RESTARTS   AGE   IP                NODE     NOMINATED NODE   READINESS GATES
kube-system   canal-j458h                                     2/2     Running     0          68m   192.168.121.62    clr-01   <none>           <none>
kube-system   canal-jh2xp                                     2/2     Running     0          62m   192.168.121.147   clr-02   <none>           <none>
kube-system   canal-kghpl                                     2/2     Running     0          76m   192.168.121.40    clr-03   <none>           <none>
kube-system   coredns-5644d7b6d9-qnjpj                        1/1     Running     0          76m   10.244.0.12       clr-03   <none>           <none>
kube-system   coredns-5644d7b6d9-szwpl                        1/1     Running     0          76m   10.244.0.5        clr-03   <none>           <none>
kube-system   etcd-clr-03                                     1/1     Running     0          74m   192.168.121.40    clr-03   <none>           <none>
kube-system   kata-deploy-6gmb8                               1/1     Running     0          61m   10.244.2.3        clr-02   <none>           <none>
kube-system   kata-deploy-csb54                               1/1     Running     0          75m   10.244.0.3        clr-03   <none>           <none>
kube-system   kata-deploy-kqd82                               1/1     Running     0          67m   10.244.1.2        clr-01   <none>           <none>
kube-system   kube-apiserver-clr-03                           1/1     Running     0          76m   192.168.121.40    clr-03   <none>           <none>
kube-system   kube-controller-manager-clr-03                  1/1     Running     0          74m   192.168.121.40    clr-03   <none>           <none>
kube-system   kube-proxy-f4vsm                                1/1     Running     0          68m   192.168.121.62    clr-01   <none>           <none>
kube-system   kube-proxy-jhvr2                                1/1     Running     0          62m   192.168.121.147   clr-02   <none>           <none>
kube-system   kube-proxy-nbt2s                                1/1     Running     0          76m   192.168.121.40    clr-03   <none>           <none>
kube-system   kube-scheduler-clr-03                           1/1     Running     0          75m   192.168.121.40    clr-03   <none>           <none>
kube-system   metrics-server-69dfdf998b-64tl5                 1/1     Running     0          76m   10.244.0.10       clr-03   <none>           <none>
rook-ceph     csi-cephfsplugin-64dd4                          3/3     Running     0          61m   192.168.121.147   clr-02   <none>           <none>
rook-ceph     csi-cephfsplugin-fpg2f                          3/3     Running     0          67m   192.168.121.62    clr-01   <none>           <none>
rook-ceph     csi-cephfsplugin-provisioner-7c44c4ff49-df2pd   4/4     Running     0          75m   10.244.0.8        clr-03   <none>           <none>
rook-ceph     csi-cephfsplugin-provisioner-7c44c4ff49-qndzp   4/4     Running     0          75m   10.244.0.9        clr-03   <none>           <none>
rook-ceph     csi-cephfsplugin-xmc2t                          3/3     Running     0          75m   192.168.121.40    clr-03   <none>           <none>
rook-ceph     csi-rbdplugin-g6vcn                             3/3     Running     0          75m   192.168.121.40    clr-03   <none>           <none>
rook-ceph     csi-rbdplugin-provisioner-7458d98547-dlpkr      5/5     Running     0          75m   10.244.0.7        clr-03   <none>           <none>
rook-ceph     csi-rbdplugin-provisioner-7458d98547-kz69v      5/5     Running     1          75m   10.244.0.6        clr-03   <none>           <none>
rook-ceph     csi-rbdplugin-pwnh5                             3/3     Running     0          67m   192.168.121.62    clr-01   <none>           <none>
rook-ceph     csi-rbdplugin-qdcqg                             3/3     Running     0          61m   192.168.121.147   clr-02   <none>           <none>
rook-ceph     rook-ceph-mgr-a-845d97f7c5-nl564                1/1     Running     0          59m   10.244.1.6        clr-01   <none>           <none>
rook-ceph     rook-ceph-mon-a-55b7cc7d7f-4n7f4                1/1     Running     0          61m   10.244.0.15       clr-03   <none>           <none>
rook-ceph     rook-ceph-mon-b-74b8ddc886-hdf7m                1/1     Running     0          61m   10.244.1.5        clr-01   <none>           <none>
rook-ceph     rook-ceph-mon-c-7678967748-9fc4b                1/1     Running     0          60m   10.244.2.5        clr-02   <none>           <none>
rook-ceph     rook-ceph-operator-75d95cb868-86sjz             1/1     Running     0          75m   10.244.0.2        clr-03   <none>           <none>
rook-ceph     rook-ceph-osd-0-78b864678c-z2mng                1/1     Running     0          57m   10.244.1.9        clr-01   <none>           <none>
rook-ceph     rook-ceph-osd-1-5894b9f5d6-5zggx                1/1     Running     0          57m   10.244.2.7        clr-02   <none>           <none>
rook-ceph     rook-ceph-osd-2-67dc44f97-6tlz6                 1/1     Running     0          56m   10.244.0.19       clr-03   <none>           <none>
rook-ceph     rook-ceph-osd-3-565cbf8cf4-gtcgm                1/1     Running     0          57m   10.244.1.8        clr-01   <none>           <none>
rook-ceph     rook-ceph-osd-4-789dc7b77c-j4568                1/1     Running     0          57m   10.244.2.9        clr-02   <none>           <none>
rook-ceph     rook-ceph-osd-5-b7d87684d-4lfcf                 1/1     Running     0          56m   10.244.0.18       clr-03   <none>           <none>
rook-ceph     rook-ceph-osd-6-774c7bfd4c-kk5d6                1/1     Running     0          57m   10.244.1.10       clr-01   <none>           <none>
rook-ceph     rook-ceph-osd-7-896657cf8-96f5f                 1/1     Running     0          57m   10.244.2.8        clr-02   <none>           <none>
rook-ceph     rook-ceph-osd-8-6998bccd8f-llcdt                1/1     Running     0          56m   10.244.0.17       clr-03   <none>           <none>
rook-ceph     rook-ceph-osd-prepare-clr-01-j9hwq              0/1     Completed   0          58m   10.244.1.7        clr-01   <none>           <none>
rook-ceph     rook-ceph-osd-prepare-clr-02-qwbkt              0/1     Completed   0          58m   10.244.2.6        clr-02   <none>           <none>
rook-ceph     rook-ceph-osd-prepare-clr-03-jdqtg              0/1     Completed   0          58m   10.244.0.16       clr-03   <none>           <none>
rook-ceph     rook-discover-j2pzk                             1/1     Running     0          75m   10.244.0.4        clr-03   <none>           <none>
rook-ceph     rook-discover-rswz8                             1/1     Running     0          61m   10.244.2.2        clr-02   <none>           <none>
rook-ceph     rook-discover-wq5nh                             1/1     Running     0          67m   10.244.1.3        clr-01   <none>           <none>

@knangia
Copy link
Contributor

knangia commented Oct 5, 2019 •
edited
Loading

Will be testing with runc version 1.0.0-rc8 and will update.

@NitinAtIntel
Copy link
Contributor

Will be testing with runc version 1.0.0-rc8 and will update.

Thanks @knangia , it is worth noting here that runc 1.0.0-rc8 is not yet upstream in Clear. WIP and absolutely worth checking.

@NitinAtIntel
Copy link
Contributor

runc 1.0.0-rc8 is upstream in Clear.

@knangia
Copy link
Contributor

knangia commented Oct 24, 2019

All pods come up and running, tested with runc version 1.0.0-rc8 and Kubernetes v1.16.2

clear@clr-01~ $ kubectl get po -A -o wide
NAMESPACE              NAME                                            READY   STATUS      RESTARTS   AGE     IP                NODE     NOMINATED NODE   READINESS GATES
ingress-nginx          nginx-ingress-controller-67cff8fb65-k4lb5       1/1     Running     0          3m18s   10.244.2.21       clr-03   <none>           <none>
kube-system            canal-2nqxn                                     2/2     Running     0          8m23s   192.168.121.104   clr-03   <none>           <none>
kube-system            canal-l5vtz                                     2/2     Running     0          10m     192.168.121.163   clr-02   <none>           <none>
kube-system            canal-qtpx2                                     2/2     Running     0          11m     192.168.121.36    clr-01   <none>           <none>
kube-system            coredns-5644d7b6d9-f4xmc                        1/1     Running     0          17m     10.244.0.4        clr-01   <none>           <none>
kube-system            coredns-5644d7b6d9-gshhr                        1/1     Running     0          17m     10.244.0.2        clr-01   <none>           <none>
kube-system            elasticsearch-logging-0                         1/1     Running     0          3m24s   10.244.2.19       clr-03   <none>           <none>
kube-system            elasticsearch-logging-1                         1/1     Running     0          98s     10.244.0.18       clr-01   <none>           <none>
kube-system            etcd-clr-01                                     1/1     Running     0          17m     192.168.121.36    clr-01   <none>           <none>
kube-system            fluentd-es-v2.5.2-2n54v                         1/1     Running     0          3m24s   10.244.0.17       clr-01   <none>           <none>
kube-system            fluentd-es-v2.5.2-rpgpt                         1/1     Running     0          3m24s   10.244.2.20       clr-03   <none>           <none>
kube-system            fluentd-es-v2.5.2-wvwjp                         1/1     Running     0          3m24s   10.244.1.15       clr-02   <none>           <none>
kube-system            kata-deploy-55hg8                               1/1     Running     0          10m     10.244.0.6        clr-01   <none>           <none>
kube-system            kata-deploy-65495                               1/1     Running     0          8m3s    10.244.2.3        clr-03   <none>           <none>
kube-system            kata-deploy-92l5v                               1/1     Running     0          10m     10.244.1.2        clr-02   <none>           <none>
kube-system            kibana-logging-7f6b4b96b4-fv9c7                 1/1     Running     0          3m24s   10.244.2.18       clr-03   <none>           <none>
kube-system            kube-apiserver-clr-01                           1/1     Running     0          17m     192.168.121.36    clr-01   <none>           <none>
kube-system            kube-controller-manager-clr-01                  1/1     Running     0          17m     192.168.121.36    clr-01   <none>           <none>
kube-system            kube-proxy-8c6t4                                1/1     Running     0          10m     192.168.121.163   clr-02   <none>           <none>
kube-system            kube-proxy-nr5kf                                1/1     Running     0          17m     192.168.121.36    clr-01   <none>           <none>
kube-system            kube-proxy-qrcxp                                1/1     Running     0          8m23s   192.168.121.104   clr-03   <none>           <none>
kube-system            kube-scheduler-clr-01                           1/1     Running     0          17m     192.168.121.36    clr-01   <none>           <none>
kube-system            metrics-server-69dfdf998b-m7c6z                 1/1     Running     0          11m     10.244.0.3        clr-01   <none>           <none>
kubernetes-dashboard   dashboard-metrics-scraper-6ccf7f6cc8-68zwx      1/1     Running     0          4m14s   10.244.2.14       clr-03   <none>           <none>
kubernetes-dashboard   kubernetes-dashboard-6865b885fc-vj4b4           1/1     Running     0          4m14s   10.244.2.15       clr-03   <none>           <none>
monitoring             alertmanager-main-0                             2/2     Running     0          3m45s   10.244.2.16       clr-03   <none>           <none>
monitoring             alertmanager-main-1                             2/2     Running     0          3m45s   10.244.1.13       clr-02   <none>           <none>
monitoring             alertmanager-main-2                             2/2     Running     0          3m45s   10.244.0.16       clr-01   <none>           <none>
monitoring             grafana-5cd56df4cd-jb7q9                        1/1     Running     0          4m21s   10.244.2.10       clr-03   <none>           <none>
monitoring             kube-state-metrics-79d4b9b497-d2rxd             3/3     Running     0          4m21s   10.244.2.11       clr-03   <none>           <none>
monitoring             node-exporter-7ggjp                             2/2     Running     0          4m21s   192.168.121.36    clr-01   <none>           <none>
monitoring             node-exporter-bm4wh                             2/2     Running     0          4m21s   192.168.121.163   clr-02   <none>           <none>
monitoring             node-exporter-xhkkk                             2/2     Running     0          4m21s   192.168.121.104   clr-03   <none>           <none>
monitoring             prometheus-adapter-c676d8764-rtnbr              1/1     Running     0          4m21s   10.244.2.12       clr-03   <none>           <none>
monitoring             prometheus-k8s-0                                3/3     Running     1          3m35s   10.244.2.17       clr-03   <none>           <none>
monitoring             prometheus-k8s-1                                3/3     Running     1          3m35s   10.244.1.14       clr-02   <none>           <none>
monitoring             prometheus-operator-7559d67ff-chvlj             1/1     Running     0          4m21s   10.244.2.13       clr-03   <none>           <none>
rook-ceph              csi-cephfsplugin-94sp2                          3/3     Running     0          9m43s   192.168.121.163   clr-02   <none>           <none>
rook-ceph              csi-cephfsplugin-provisioner-7c44c4ff49-ft878   4/4     Running     0          9m43s   10.244.1.4        clr-02   <none>           <none>
rook-ceph              csi-cephfsplugin-provisioner-7c44c4ff49-qx24x   4/4     Running     0          9m43s   10.244.0.8        clr-01   <none>           <none>
rook-ceph              csi-cephfsplugin-rhbp9                          3/3     Running     0          9m43s   192.168.121.36    clr-01   <none>           <none>
rook-ceph              csi-cephfsplugin-xltp9                          3/3     Running     0          8m3s    192.168.121.104   clr-03   <none>           <none>
rook-ceph              csi-rbdplugin-5mbdm                             3/3     Running     0          9m43s   192.168.121.36    clr-01   <none>           <none>
rook-ceph              csi-rbdplugin-jwwz4                             3/3     Running     0          8m3s    192.168.121.104   clr-03   <none>           <none>
rook-ceph              csi-rbdplugin-provisioner-7458d98547-k4ngm      5/5     Running     0          9m43s   10.244.1.5        clr-02   <none>           <none>
rook-ceph              csi-rbdplugin-provisioner-7458d98547-pnsj2      5/5     Running     0          9m43s   10.244.0.9        clr-01   <none>           <none>
rook-ceph              csi-rbdplugin-vt6r5                             3/3     Running     0          9m43s   192.168.121.163   clr-02   <none>           <none>
rook-ceph              rook-ceph-mgr-a-69df7f8fd4-stbsr                1/1     Running     0          6m20s   10.244.2.5        clr-03   <none>           <none>
rook-ceph              rook-ceph-mon-a-6fcc96d456-4fvf6                1/1     Running     0          7m51s   10.244.1.8        clr-02   <none>           <none>
rook-ceph              rook-ceph-mon-b-69d4448554-d2xbn                1/1     Running     0          7m37s   10.244.0.11       clr-01   <none>           <none>
rook-ceph              rook-ceph-mon-c-86674bf9f7-bmlfv                1/1     Running     0          7m18s   10.244.2.4        clr-03   <none>           <none>
rook-ceph              rook-ceph-operator-75d95cb868-xxc9x             1/1     Running     0          11m     10.244.0.5        clr-01   <none>           <none>
rook-ceph              rook-ceph-osd-0-5ff4fc6546-x2tl5                1/1     Running     0          5m17s   10.244.0.13       clr-01   <none>           <none>
rook-ceph              rook-ceph-osd-1-84c674bd75-c886k                1/1     Running     0          5m17s   10.244.1.11       clr-02   <none>           <none>
rook-ceph              rook-ceph-osd-2-76b76476f6-mfgbc                1/1     Running     0          5m17s   10.244.2.7        clr-03   <none>           <none>
rook-ceph              rook-ceph-osd-3-7f89bdf995-kmmhr                1/1     Running     0          5m17s   10.244.0.14       clr-01   <none>           <none>
rook-ceph              rook-ceph-osd-4-764c6c9c6d-v9wd4                1/1     Running     0          5m17s   10.244.1.10       clr-02   <none>           <none>
rook-ceph              rook-ceph-osd-5-547dccd96f-sxs5b                1/1     Running     0          5m17s   10.244.2.8        clr-03   <none>           <none>
rook-ceph              rook-ceph-osd-6-67fd698d76-mrbgx                1/1     Running     0          5m17s   10.244.1.12       clr-02   <none>           <none>
rook-ceph              rook-ceph-osd-7-64bf85db45-jpnp8                1/1     Running     0          5m17s   10.244.0.15       clr-01   <none>           <none>
rook-ceph              rook-ceph-osd-8-5fc5d9d6bb-l9rxg                1/1     Running     0          5m17s   10.244.2.9        clr-03   <none>           <none>
rook-ceph              rook-ceph-osd-prepare-clr-01-s8gql              0/1     Completed   0          5m59s   10.244.0.12       clr-01   <none>           <none>
rook-ceph              rook-ceph-osd-prepare-clr-02-sl95q              0/1     Completed   0          5m59s   10.244.1.9        clr-02   <none>           <none>
rook-ceph              rook-ceph-osd-prepare-clr-03-zn2jv              0/1     Completed   0          5m58s   10.244.2.6        clr-03   <none>           <none>
rook-ceph              rook-discover-jvj44                             1/1     Running     0          9m45s   10.244.0.7        clr-01   <none>           <none>
rook-ceph              rook-discover-qhl2b                             1/1     Running     0          9m45s   10.244.1.3        clr-02   <none>           <none>
rook-ceph              rook-discover-vpr86                             1/1     Running     0          8m3s    10.244.2.2        clr-03   <none>           <none>

@knangia
Copy link
Contributor

knangia commented Oct 24, 2019

I'd suggest closing this issue, as all pods are up and running, and don't face any cgroup related isuue.

@knangia knangia closed this as completed Oct 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@knangia knangia

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ganeshmaharaj @mcastelino @NitinAtIntel @knangia