Honor KUBERNETES_SERVICE_HOST in k8s client

[Retna-Gjensidige]

This issue is for linkerd2/policy-controller where this lib is being used.

We are testing linkerd2 stable-2.12.0 and we see that policy-controller running in the Destination pod is not able to connect to the API-server and ends up with crash/restart loop.

Our current installation with linkerd2 stable-2.11.3 all is good with the policy-controller being able to access API server.

What we see is that policy-controller is not using the KUBERNETES_SERVICE_HOST env variable to connect to the API-server. Its using kubernetes.default.svc as the url to API-server.

Would it be possible to have kubert honor KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT when communication to the API-server?

---

Logs from policy-controller container:

{"timestamp":"2022-09-02T09:28:19.075636Z","level":"DEBUG","fields":{"service.ready":true,"message":"processing request"},"target":"tower::buffer::worker","spans":[{"name":"networkauthentications"}]}
{"timestamp":"2022-09-02T09:28:19.075645Z","level":"DEBUG","fields":{"service.ready":true,"message":"processing request"},"target":"tower::buffer::worker","spans":[{"name":"httproutes"}]}
{"timestamp":"2022-09-02T09:28:19.075706Z","level":"DEBUG","fields":{"message":"requesting"},"target":"kube_client::client::builder","spans":[{"name":"httproutes"},{"http.method":"GET","http.url":"https://kubernetes.default.svc/apis/policy.linkerd.io/v1alpha1/httproutes?","otel.kind":"client","otel.name":"list","name":"HTTP"}]}
{"timestamp":"2022-09-02T09:28:19.075820Z","level":"DEBUG","fields":{"message":"requesting"},"target":"kube_client::client::builder","spans":[{"name":"meshtlsauthentications"},{"http.method":"GET","http.url":"https://kubernetes.default.svc/apis/policy.linkerd.io/v1alpha1/meshtlsauthentications?","otel.kind":"client","otel.name":"list","name":"HTTP"}]}
{"timestamp":"2022-09-02T09:28:19.075699Z","level":"DEBUG","fields":{"message":"resolving host=\"kubernetes.default.svc\""},"target":"hyper::client::connect::dns"}
{"timestamp":"2022-09-02T09:28:19.077097Z","level":"DEBUG","fields":{"message":"requesting"},"target":"kube_client::client::builder","spans":[{"name":"serverauthorizations"},{"http.method":"GET","http.url":"https://kubernetes.default.svc/apis/policy.linkerd.io/v1beta1/serverauthorizations?","otel.kind":"client","otel.name":"list","name":"HTTP"}]}
{"timestamp":"2022-09-02T09:28:19.077093Z","level":"DEBUG","fields":{"message":"requesting"},"target":"kube_client::client::builder","spans":[{"name":"networkauthentications"},{"http.method":"GET","http.url":"https://kubernetes.default.svc/apis/policy.linkerd.io/v1alpha1/networkauthentications?","otel.kind":"client","otel.name":"list","name":"HTTP"}]}
{"timestamp":"2022-09-02T09:28:19.078258Z","level":"DEBUG","fields":{"message":"resolving host=\"kubernetes.default.svc\""},"target":"hyper::client::connect::dns"}
{"timestamp":"2022-09-02T09:28:19.079550Z","level":"DEBUG","fields":{"message":"resolving host=\"kubernetes.default.svc\""},"target":"hyper::client::connect::dns"}
{"timestamp":"2022-09-02T09:28:19.078270Z","level":"DEBUG","fields":{"message":"resolving host=\"kubernetes.default.svc\""},"target":"hyper::client::connect::dns"}
{"timestamp":"2022-09-02T09:28:19.079722Z","level":"DEBUG","fields":{"message":"connecting to 10.2.0.1:443"},"target":"hyper::client::connect::http","spans":[{"name":"pods"},{"http.method":"GET","http.url":"https://kubernetes.default.svc/api/v1/pods?&labelSelector=linkerd.io%2Fcontrol-plane-ns","otel.kind":"client","otel.name":"list","name":"HTTP"}]}
{"timestamp":"2022-09-02T09:28:19.079751Z","level":"DEBUG","fields":{"message":"connecting to 10.2.0.1:443"},"target":"hyper::client::connect::http","spans":[{"name":"authorizationpolicies"},{"http.method":"GET","http.url":"https://kubernetes.default.svc/apis/policy.linkerd.io/v1alpha1/authorizationpolicies?","otel.kind":"client","otel.name":"list","name":"HTTP"}]}
{"timestamp":"2022-09-02T09:28:19.078340Z","level":"DEBUG","fields":{"message":"resolving host=\"kubernetes.default.svc\""},"target":"hyper::client::connect::dns"}
{"timestamp":"2022-09-02T09:28:19.079558Z","level":"DEBUG","fields":{"message":"resolving host=\"kubernetes.default.svc\""},"target":"hyper::client::connect::dns"}
{"timestamp":"2022-09-02T09:28:19.081938Z","level":"DEBUG","fields":{"message":"connecting to 10.2.0.1:443"},"target":"hyper::client::connect::http","spans":[{"name":"serverauthorizations"},{"http.method":"GET","http.url":"https://kubernetes.default.svc/apis/policy.linkerd.io/v1beta1/serverauthorizations?","otel.kind":"client","otel.name":"list","name":"HTTP"}]}
{"timestamp":"2022-09-02T09:28:19.082054Z","level":"DEBUG","fields":{"message":"connecting to 10.2.0.1:443"},"target":"hyper::client::connect::http","spans":[{"name":"networkauthentications"},{"http.method":"GET","http.url":"https://kubernetes.default.svc/apis/policy.linkerd.io/v1alpha1/networkauthentications?","otel.kind":"client","otel.name":"list","name":"HTTP"}]}
{"timestamp":"2022-09-02T09:28:19.082074Z","level":"DEBUG","fields":{"message":"connecting to 10.2.0.1:443"},"target":"hyper::client::connect::http","spans":[{"name":"httproutes"},{"http.method":"GET","http.url":"https://kubernetes.default.svc/apis/policy.linkerd.io/v1alpha1/httproutes?","otel.kind":"client","otel.name":"list","name":"HTTP"}]}
{"timestamp":"2022-09-02T09:28:19.083221Z","level":"DEBUG","fields":{"message":"connected to 10.2.0.1:443"},"target":"hyper::client::connect::http","spans":[{"name":"pods"},{"http.method":"GET","http.url":"https://kubernetes.default.svc/api/v1/pods?&labelSelector=linkerd.io%2Fcontrol-plane-ns","otel.kind":"client","otel.name":"list","name":"HTTP"}]}
{"timestamp":"2022-09-02T09:28:19.083242Z","level":"DEBUG","fields":{"message":"connected to 10.2.0.1:443"},"target":"hyper::client::connect::http","spans":[{"name":"serverauthorizations"},{"http.method":"GET","http.url":"https://kubernetes.default.svc/apis/policy.linkerd.io/v1beta1/serverauthorizations?","otel.kind":"client","otel.name":"list","name":"HTTP"}]}
{"timestamp":"2022-09-02T09:28:19.083344Z","level":"DEBUG","fields":{"message":"connected to 10.2.0.1:443"},"target":"hyper::client::connect::http","spans":[{"name":"authorizationpolicies"},{"http.method":"GET","http.url":"https://kubernetes.default.svc/apis/policy.linkerd.io/v1alpha1/authorizationpolicies?","otel.kind":"client","otel.name":"list","name":"HTTP"}]}
{"timestamp":"2022-09-02T09:28:19.083535Z","level":"DEBUG","fields":{"message":"connecting to 10.2.0.1:443"},"target":"hyper::client::connect::http","spans":[{"name":"servers"},{"http.method":"GET","http.url":"https://kubernetes.default.svc/apis/policy.linkerd.io/v1beta1/servers?","otel.kind":"client","otel.name":"list","name":"HTTP"}]}
{"timestamp":"2022-09-02T09:28:19.083555Z","level":"DEBUG","fields":{"message":"connected to 10.2.0.1:443"},"target":"hyper::client::connect::http","spans":[{"name":"networkauthentications"},{"http.method":"GET","http.url":"https://kubernetes.default.svc/apis/policy.linkerd.io/v1alpha1/networkauthentications?","otel.kind":"client","otel.name":"list","name":"HTTP"}]}
{"timestamp":"2022-09-02T09:28:19.083571Z","level":"DEBUG","fields":{"message":"connected to 10.2.0.1:443"},"target":"hyper::client::connect::http","spans":[{"name":"httproutes"},{"http.method":"GET","http.url":"https://kubernetes.default.svc/apis/policy.linkerd.io/v1alpha1/httproutes?","otel.kind":"client","otel.name":"list","name":"HTTP"}]}
{"timestamp":"2022-09-02T09:28:19.083901Z","level":"DEBUG","fields":{"message":"connecting to 10.2.0.1:443"},"target":"hyper::client::connect::http","spans":[{"name":"meshtlsauthentications"},{"http.method":"GET","http.url":"https://kubernetes.default.svc/apis/policy.linkerd.io/v1alpha1/meshtlsauthentications?","otel.kind":"client","otel.name":"list","name":"HTTP"}]}
{"timestamp":"2022-09-02T09:28:19.085181Z","level":"DEBUG","fields":{"message":"connected to 10.2.0.1:443"},"target":"hyper::client::connect::http","spans":[{"name":"meshtlsauthentications"},{"http.method":"GET","http.url":"https://kubernetes.default.svc/apis/policy.linkerd.io/v1alpha1/meshtlsauthentications?","otel.kind":"client","otel.name":"list","name":"HTTP"}]}
{"timestamp":"2022-09-02T09:28:19.085214Z","level":"ERROR","fields":{"message":"failed with error error trying to connect: unexpected EOF"},"target":"kube_client::client::builder","spans":[{"name":"serverauthorizations"},{"http.method":"GET","http.url":"https://kubernetes.default.svc/apis/policy.linkerd.io/v1beta1/serverauthorizations?","otel.kind":"client","otel.name":"list","otel.status_code":"ERROR","name":"HTTP"}]}
{"timestamp":"2022-09-02T09:28:19.085219Z","level":"ERROR","fields":{"message":"failed with error error trying to connect: unexpected EOF"},"target":"kube_client::client::builder","spans":[{"name":"pods"},{"http.method":"GET","http.url":"https://kubernetes.default.svc/api/v1/pods?&labelSelector=linkerd.io%2Fcontrol-plane-ns","otel.kind":"client","otel.name":"list","otel.status_code":"ERROR","name":"HTTP"}]}
{"timestamp":"2022-09-02T09:28:19.085257Z","level":"INFO","fields":{"message":"stream failed","error":"failed to perform initial object list: HyperError: error trying to connect: unexpected EOF"},"target":"kubert::errors","spans":[{"name":"serverauthorizations"}]}
{"timestamp":"2022-09-02T09:28:19.085283Z","level":"INFO","fields":{"message":"stream failed","error":"failed to perform initial object list: HyperError: error trying to connect: unexpected EOF"},"target":"kubert::errors","spans":[{"name":"pods"}]}

[olix0r]

@Retna-Gjensidige Interesting. I'm not opposed to this at all, but I'm also not convinced that this will address your issues.

We are testing linkerd2 stable-2.12.0 and we see that policy-controller running in the Destination pod is not able to connect to the API-server and ends up with crash/restart loop.

Our current installation with linkerd2 stable-2.11.3 all is good with the policy-controller being able to access API server.

As far as I understand, nothing has changed with regard to how kubert or the policy controller discovers the Kubernetes API between these releases.

Does the kubernetes service not exist in the default namespace on your cluster? It looks like we're resolving it to addresses and attempting to connect to it.

Is this related to an open Linkerd2 issue? It would be helpful to have some more details about your cluster configuration so we can be sure that we're working on the right thing.

[olix0r]

If this change is indeed needed, it's probably not appropriate to make this change in this repo. We're loading our kubernetes client configuration via https://github.com/kube-rs/kube-rs/blob/050bf9d23e32af9c1f65b3a01dda6f677e64055b/kube-client/src/config/mod.rs#L209-L235

[olix0r]

I'm going to close this issue, since this repo isn't the right place to address this problem. If you can get as many details as possible in an issue in the Linkerd2 repo, we can help narrow down the source of the problem and identify the appropriate fix. Thanks!

[olix0r]

Note that the linked kube-rs code references https://kubernetes.io/docs/tasks/run-application/access-api-from-pod/#directly-accessing-the-rest-api, which says:

While running in a Pod, the Kubernetes apiserver is accessible via a Service named kubernetes in the default namespace. Therefore, Pods can use the kubernetes.default.svc hostname to query the API server. Official client libraries do this automatically.

[olix0r]

And note that support for these environment variables was explicitly removed from kube-rs in kube-rs/kube@dd0b258, as the (legacy) environment-based method is buggy.

[olix0r]

As far as I understand, nothing has changed with regard to how kubert or the policy controller discovers the Kubernetes API between these releases.

Well, I guess we clarified this: kube-rs/kube@dd0b258 happened. It's still worth getting a Linkerd2 issue so that others can get visibility into the issue you're seeing. It would be helpful to understand how your cluster setup diverges from the standard setup where the cluster-local service can be used. Then we can help engage the kube-rs maintainers to, perhaps, restore support for systems that don't support the standard access mechanism.

[Retna-Gjensidige]

@olix0r Thanks for taking the time to dig into the issue ❤️. Our requirement may be specific to Azure as we secure our AKS egress with a layer 7 firewall.

Info here, plus Azure AKS has also support for the KUBERNETES_SERVICE_HOST now, release notes here

We will address this issue in Linkerd2 repo as you suggested 🙏