Resolve spec.apiServer from kubectl context at runtime

[igorwwwwwwwwwwwwwwwwwwww]

One of the issues we're facing at @GitLab is apiServer being hardcoded in a shared spec.json file used for minikube.

It is possible to set this value automatically using:

tk env set environments/jaeger/dev --server-from-context minikube

However, this will still update the file. So it requires a non-shared (possibly copy-pasted from spec.json.example) spec file, so that users don't keep overriding each other.

Some of the suggestions in #390 may help with this.

But I was wondering if it might make sense for tanka to read kubectl config get-contexts and figure out the apiServer dynamically that way at runtime.

[issue-label-bot]

Issue-Label Bot is automatically applying the label kind/feature to this issue, with a confidence of 0.71. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[malcolmholmes]

Thanks for this question. The reason we store the api server in the config is because it is far better for identifying the cluster than a context name. Individual users could name contexts differently, whereas the api server URL will be consistently correct across users. This helps us guarantee that when we do a tk apply, the correct cluster will be used. This behaviour is intentional, and we don't want to change it.

However, the minikube situation is also a valid one - when the same config might be applied to multiple local clusters, with different API server URLs and likely different context names. To meet this case, we'd need to find something that allows supporting minikube, but doesn't risk people using context names for their production clusters.

[malcolmholmes]

The basic idea as I see it is that it needs to be easy to specify the cluster api server URL, and hard to configure a context (but still possible). If we can get that in place, we could have something that works.

[sh0rez]

Hi @igorwwwwwwwwwwwwwwwwwwww! As @malcolmholmes already explained, we chose operational safety over ease of use here back when we designed thius.

As we have been playing with the idea of moving spec.json into Jsonnet itself a lot recently, I think the solution tho this could like within there. Using a combination of advanced Jsonnet concepts, this could certainly be achieved.

Say the env config in Jsonnet would like this:

{
  env: {
    apiVersion: "tanka.dev/v1alpha1",
    kind: "Environment",
    metadata: { name: "/environments/myFancyEnv" },
    spec: {
      apiServer: "https://myFancyCluster:6443",
    }
  }
}

Top level arguments would allow you to dynamically adapt the apiServer url:

-{
+function(apiServer="https://myFancyCluster:6443") {
   env: {
     apiVersion: "tanka.dev/v1alpha1",
     kind: "Environment",
     metadata: { name: "/environments/myFancyEnv" },
     spec: {
-      apiServer: "https://myFancyCluster:6443",
+      apiServer: apiServer,
     }
   }
 }

When invoked without arguments, default value of https://myFancyCluster:6443 is used:

$ tk show environments/myFancyEnv

However, you can now inject this value:

$ tk show environments/myFancyEnv --tla-str "apiServer=https://$(minikube ip):6443"

How does this sound? This should be sufficiently difficult to avoid people doing it automatically, but a suitable solution for dev servers.

[igorwwwwwwwwwwwwwwwwwwww]

@sh0rez Nice! Yes, I think this would work for us. 👍

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Duologic]

Inline environments is available in v0.13.0, does this resolve your question?

[sh0rez]

The solution outlined in #394 (comment) became possible in #403 thanks to @Duologic. Closing as resolved

[James-Riordan]

Hi @igorwwwwwwwwwwwwwwwwwwww! As @malcolmholmes already explained, we chose operational safety over ease of use here back when we designed thius.

As we have been playing with the idea of moving spec.json into Jsonnet itself a lot recently, I think the solution tho this could like within there. Using a combination of advanced Jsonnet concepts, this could certainly be achieved.

Say the env config in Jsonnet would like this:

{
  env: {
    apiVersion: "tanka.dev/v1alpha1",
    kind: "Environment",
    metadata: { name: "/environments/myFancyEnv" },
    spec: {
      apiServer: "https://myFancyCluster:6443",
    }
  }
}

Top level arguments would allow you to dynamically adapt the apiServer url:

-{
+function(apiServer="https://myFancyCluster:6443") {
   env: {
     apiVersion: "tanka.dev/v1alpha1",
     kind: "Environment",
     metadata: { name: "/environments/myFancyEnv" },
     spec: {
-      apiServer: "https://myFancyCluster:6443",
+      apiServer: apiServer,
     }
   }
 }

When invoked without arguments, default value of https://myFancyCluster:6443 is used:

$ tk show environments/myFancyEnv

However, you can now inject this value:

$ tk show environments/myFancyEnv --tla-str "apiServer=https://$(minikube ip):6443"

How does this sound? This should be sufficiently difficult to avoid people doing it automatically, but a suitable solution for dev servers.

Can we please make spec.json -> spec.jsonnet? The old paradigm of "one-time create k8s clusters" is no longer the case. K8s clusters should be assumed ephemeral, and thus the dynamic apiServer seems naturally fitting for the evolution of K8s over the years.

[Duologic]

Have a look at this, should cover your request: https://tanka.dev/inline-environments