" the server has asked for the client to provide credentials "

[gamename]

Hi,

I'm getting this error about every other time I start:

time="2017-10-09T17:18:27Z" level=info msg="Scanning plugins in ./plugins.d (pwd: /)"
time="2017-10-09T17:18:27Z" level=warning msg="Unsupported plugin file detected ..109810_09_10_17_18_23.899671353"
time="2017-10-09T17:18:27Z" level=warning msg="Unsupported plugin file detected ..data"
time="2017-10-09T17:18:27Z" level=info msg="Scanning plugins in /etc/sonobuoy/plugins.d (pwd: /)"
time="2017-10-09T17:18:27Z" level=info msg="Directory (/etc/sonobuoy/plugins.d) does not exist"
time="2017-10-09T17:18:27Z" level=info msg="Scanning plugins in ~/sonobuoy/plugins.d (pwd: /)"
time="2017-10-09T17:18:27Z" level=info msg="Directory (~/sonobuoy/plugins.d) does not exist"
time="2017-10-09T17:18:27Z" level=info msg="Loading plugin driver DaemonSet"
time="2017-10-09T17:18:27Z" level=info msg="Loading plugin driver Job"
panic: the server has asked for the client to provide credentials (get namespaces)

goroutine 1 [running]:
github.com/heptio/sonobuoy/pkg/discovery.FilterNamespaces(0x1883b00, 0xc420328dd0, 0xc4202feeda, 0x2, 0x3f, 0x0, 0x0)
	/go/src/github.com/heptio/sonobuoy/pkg/discovery/utils.go:44 +0x48b
github.com/heptio/sonobuoy/pkg/discovery.Run(0x1883b00, 0xc420328dd0, 0xc4201c6c80, 0x0)
	/go/src/github.com/heptio/sonobuoy/pkg/discovery/discovery.go:74 +0x45c
github.com/heptio/sonobuoy/cmd/sonobuoy/app.runMaster(0xc420234b40, 0xc4202e2880, 0x0, 0x4)
	/go/src/github.com/heptio/sonobuoy/cmd/sonobuoy/app/master.go:63 +0x8a
github.com/heptio/sonobuoy/vendor/github.com/spf13/cobra.(*Command).execute(0xc420234b40, 0xc4202e2780, 0x4, 0x4, 0xc420234b40, 0xc4202e2780)
	/go/src/github.com/heptio/sonobuoy/vendor/github.com/spf13/cobra/command.go:654 +0x299
github.com/heptio/sonobuoy/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x18c5560, 0x1, 0x0, 0x0)
	/go/src/github.com/heptio/sonobuoy/vendor/github.com/spf13/cobra/command.go:729 +0x339
github.com/heptio/sonobuoy/vendor/github.com/spf13/cobra.(*Command).Execute(0x18c5560, 0xc4200001a0, 0xc4200001a0)
	/go/src/github.com/heptio/sonobuoy/vendor/github.com/spf13/cobra/command.go:688 +0x2b
main.main()
	/go/src/github.com/heptio/sonobuoy/cmd/sonobuoy/main.go:25 +0x2d

If I immediately stop sonobuoy and then restart it, everything starts fine.

[gamename]

kubectl --version
Kubernetes v1.7.7+coreos.0

[gamename]

Maybe this is a different bug, or maybe it isn't. But when sonobuoy appears to start ok, it still has problems:

NAMESPACE         NAME                                                                  READY     STATUS             RESTARTS   AGE       IP              NODE
default           cluster-autoscaler-4268175986-33rbg                                   1/1       Running            0          1d        10.234.16.8     ip-10-223-11-224.us-west-2.compute.internal
heptio-sonobuoy   sonobuoy                                                              1/1       Running            0          5m        10.234.16.3     ip-10-223-11-224.us-west-2.compute.internal
heptio-sonobuoy   sonobuoy-systemd-logs-daemonset-aec1116a05b2441c-glrw8                1/2       CrashLoopBackOff   5          4m        10.223.11.224   ip-10-223-11-224.us-west-2.compute.internal
heptio-sonobuoy   sonobuoy-systemd-logs-daemonset-aec1116a05b2441c-m8vfg                1/2       CrashLoopBackOff   5          4m        10.223.3.188    ip-10-223-3-188.us-west-2.compute.internal
heptio-sonobuoy   sonobuoy-systemd-logs-daemonset-aec1116a05b2441c-phjv2                1/2       CrashLoopBackOff   5          4m        10.223.5.149    ip-10-223-5-149.us-west-2.compute.internal
heptio-sonobuoy   sonobuoy-systemd-logs-daemonset-aec1116a05b2441c-pqjs6                1/2       CrashLoopBackOff   5          4m        10.223.11.140   ip-10-223-11-140.us-west-2.compute.internal
heptio-sonobuoy   sonobuoy-systemd-logs-daemonset-aec1116a05b2441c-vfc35                1/2       CrashLoopBackOff   5          4m        10.223.3.53     ip-10-223-3-53.us-west-2.compute.internal
kube-system       default-http-backend-8q8gm                                            1/1       Running            0          10m       10.234.87.8     ip-10-223-5-149.us-west-2.compute.internal

A look at one of the logs yields the following:

 kubectl logs sonobuoy-systemd-logs-daemonset-aec1116a05b2441c-glrw8 -n heptio-sonobuoy -c sonobuoy-worker
time="2017-10-09T17:27:32Z" level=info msg="Waiting on: (/tmp/results/done)"
time="2017-10-09T17:27:32Z" level=info msg="Detected done file, transmitting: (/tmp/results/systemd_logs)"
time="2017-10-09T17:27:34Z" level=error msg="error dialing master at http://sonobuoy-master:8080/api/v1/results/by-node/ip-10-223-11-224.us-west-2.compute.internal/systemd_logs: Put http://sonobuoy-master:8080/api/v1/results/by-node/ip-10-223-11-224.us-west-2.compute.internal/systemd_logs: dial tcp: lookup sonobuoy-master on 10.233.0.3:53: no such host"

[timothysc]

@gamename thx for reporting!

I'll dig more in the a.m. I was out today

[timothysc]

This is a different issue, and I've never seen it before. The top level panic is a failure to list namespaces...

        nslist, err := kubeClient.CoreV1().Namespaces().List(metav1.ListOptions{})
	if err == nil {
		for _, ns := range nslist.Items {
			logrus.Infof("Namespace %v Matched=%v", ns.Name, re.MatchString(ns.Name))
			if re.MatchString(ns.Name) {
				validns = append(validns, ns.Name)
			}
		}
	} else {
		panic(err.Error())
	}

The RBAC rules for sonobuoy are wide open.

@gamename you just ran
kubectl apply -f examples/quickstart.yaml ?

/cc @mattmoyer In case you're aware of anything related to this error "the server has asked for the client to provide credentials (get namespaces)"

[timothysc]

@gamename How do you have auth jigg'd in your environment ? It looks like a configuration issue on 1.7. IIRC you were originally running 1.6 clusters.

xref: kubernetes/kubernetes#50449 kubernetes/dashboard#374 (comment)

[gamename]

@timothysc Yes, I originally had a 1.6 environment. I ran into this error when upgrading it to 1.7.7 in a development environment.

[timothysc]

@gamename from all my google searching the signs point to api-server auth config, but I'll leave this open until we find the root cause.

[gamename]

@timothysc No worries. Just wanted to make sure I documented it in case it was something really catastrophic. Thanks for the help.

[timothysc]

I'll de-scope from 0.9.0 but leave open in case someone stumbles upon from a google search.

[timothysc]

I'm going to close out this issue, b/c we have not seen it or have a repro for 1.8+

[psaywan]

kubectl apply -f aws-auth-cm.yaml
error: You must be logged in to the server (the server has asked for the client to provide credentials)
Can anyone help

[kuchaguangjie]

The issue still remains in minikube v1.13.4.

[johnSchnake]

@kuchaguangjie Can you provide the sonobuoy version and k8s version as well so that I can try and repro?

[kuchaguangjie]

@johnSchnake I am using minikube 1.13.4. Maybe this is not relevant to this project directly, just get here after google search, without noticing the project name ...

---

Anyway following are the issue

I succeed to start a replicaset, but failed to start the service.

svc.yml:

kind: Service
apiVersion: v1
metadata:
  name: hello-svc
spec:
  selector:
    app: hello-world
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 30002

Here is the command:

kubectl create -f svc.yml

And, here is the out:

error: You must be logged in to the server (the server has asked for the client to provide credentials)

Please ignore it if you think this is not relevant to this post.

[johnSchnake]

Thanks @kuchaguangjie for the extra info. That looks like something that is just minikube specific since you aren't even using Sonobuoy at all.

You should be able to find a solution by doing a bit more googling; it seems to be a common thing for some folks depending on how they set up their cluster.

Someone created a k8s ticket as well which was closed since it seemed to be a configuration problem about the certs used: kubernetes/kubernetes#63128

Here is a minikube issue (also closed) that lists 2 different fixes that worked in different situations: kubernetes/minikube#319