monitoring-grafana pod CrashLoopBackOff

[jmgao1983]

I have a Kubernetes cluster 1.7.5, and I was trying to add Heapster following instructions at heapster1.4.2, and got this:

kubectl get pod -n kube-system -o wide
NAME                                         READY     STATUS             RESTARTS   AGE       IP            NODE
heapster-2589865292-z3jbg                    1/1       Running            0          11h       172.30.81.5   10.100.97.203
kube-dns-1575529294-wcr77                    3/3       Running            6          17h       172.30.13.3   10.100.97.205
kubernetes-dashboard-head-1964931632-clwv3   1/1       Running            0          17h       172.30.19.5   10.100.97.204
monitoring-grafana-529869525-sb0pc           0/1       CrashLoopBackOff   7          11m       172.30.55.5   10.100.97.202
monitoring-influxdb-2667871912-jkg6r         1/1       Running            0          11h       172.30.13.6   10.100.97.205

and I tried to debug using 'systemctl status kubelet.service', and found this:

helpers.go:102] Unable to get network stats from pid 23130: couldn't read network stats: failure opening /proc/23130/net/dev: open /proc/23130/net/dev: no such file or directory
helpers.go:125] Unable to get udp stats from pid 23130: failure opening /proc/23130/net/udp: open /proc/23130/net/udp: no such file or directory
helpers.go:132] Unable to get udp6 stats from pid 23130: failure opening /proc/23130/net/udp6: open /proc/23130/net/udp6: no such file or directory
...
pod_workers.go:182] Error syncing pod c41c0ce3-9838-11e7-a8c0-5254008ff8c3 ("monitoring-grafana-1880284993-p3ssx_kube-system(c41c0ce3-9838-11e7-a8c0-5254008ff8c3)"), skipping: failed to "StartContainer" for "grafana" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=grafana pod=monitoring-grafana-1880284993-p3ssx_kube-system(c41c0ce3-9838-11e7-a8c0-5254008ff8c3)"

could anyone give me some advice, thanks!

[DirectXMan12]

did you check the logs of the pod?

[jmgao1983]

@DirectXMan12
My grafana image is: gcr.io/google_containers/heapster-grafana-amd64:v4.2.0
logs

Starting a utility program that will configure Grafana
Starting Grafana in foreground mode
Can't access the Grafana dashboard. Error: Get http://admin:admin@localhost:3000/api/org: dial tcp 127.0.0.1:3000: getsockopt: connection refused. Retrying after 5 seconds...
t=2017-09-15T01:37:58+0000 lvl=crit msg="Failed to parse /etc/grafana/grafana.ini, open /etc/grafana/grafana.ini: no such file or directory%!(EXTRA []interface {}=[])"

then I tried another image:gcr.io/google_containers/heapster-grafana-amd64:v4.0.2 , and it ran up fine, could it be the image's fault?

[andyxning]

@jmgao1983

could it be the image's fault?

Yes. There is already fixed in #1709 . And, an new version for heapster-grafana-amd64 has been pushed to gcr.io and you should try heapster-grafana-amd64 with v4.4.1 and above.

[DirectXMan12]

closing issue as per the above comment