Skip to content

Latest commit

 

History

History
895 lines (758 loc) · 27.9 KB

0101-env-in-pod-template.md

File metadata and controls

895 lines (758 loc) · 27.9 KB
Raw
status title creation-date last-updated authors
implemented
Env in POD template
2022-03-17
2022-12-22
@rafalbigaj
@tomcli
@piyush-garg

TEP-0101: Env in POD template

Summary

Providing the users a functionality for specifying additional environment variables or overwriting existing environment variables defined in Steps and StepTemplate of Task at runtime through TaskRun and PipelineRun. In order to achieve that, a Pod template should support configuration of environment variables, which are combined with those defined in steps and stepTemplate, and then passed to all step containers. This let the user specify common variables to the global level as well as overwrite defaults specified on the particular step level.

Motivation

One of the most important motivators for this feature is the ability to eliminate redundant code from the PipelineRun as well as TaskRun specification. On average this can reduce 3-5 lines of yaml per each environment variable per each task in the pipeline. In case of complex pipelines, which consist of dozen or even hundreds of tasks, any kind of repetition significantly impacts the final size of PipelineRun and leads to resource exhaustion on the Kubernetes ETCD limitation.

Besides, users quite frequently are willing to overwrite environment variable values specified in a stepTemplate in the single place when running pipelines. That helps to optionally pass settings, without a need to define additional pipeline parameters.

Also, users can specify the global environment variables which needs to be added for every TaskRun or PipelineRun, then users don't need to specify them in all Tasks and changing the values of those environment variables will also be not cumbersome as they can change it at single place in global default and can also overwrite for particular run.

Having an env field in the pod template allows to:

  • specify global level defaults, what is important to reduce the size of TaskRun and PipelineRun
  • override defaults from stepTemplate at TaskRun and PipelineRun level

Goals

  1. The main goal of this proposal is to enable support for specification of environment variables at the time of execution.(TaskRun and PipelineRun)

  2. Environment variables defined in the Pod template at TaskRun and PipelineRun level take precedence over the ones defined in steps and stepTemplate.

  3. Allow cluster admin to define a list of default environment variables which gets added to all the TaskRun and PipelineRun in config-defaults.yaml

  4. Allow cluster admin to define a list of cluster-wide forbidden environment variables by providing a field in config-defaults so that users won't overwrite important environment variables configured by admin.

  5. Define the order of precedence for the multiple locations where environment variables can be defined.

Non-Goals

  1. Namespace specific environment variables are not proposed.

Use Cases

  1. In the first case, common environment variables can be defined in a single place on a PipelineRun level or TaskRun level. Values can be specified as literals or through Kubernetes references. Variables defined on a PipelineRun or TaskRun level are then available in all steps. That allows to simplify the Tekton run resource configuration and significantly reduce the size of PipelineRun and TaskRun resource, by excluding the common environment variables like static global settings, common values coming from metadata etc

  2. Secondly, environment variables defined in steps can be easily overwritten by the ones from PipelineRun and TaskRun. With that, common settings like API keys, connection details etc can be optionally overwritten in a single place.

  3. For Cloud Providers, it's very common to inject user credentials using Kubernetes API valueFrom to avoid credentials being exposed to the PodSpec. Since each cloud provider has different credential format, able to assign environment variables at the PipelineRun and TaskRun can reuse the same task with different Cloud Provider credentials. Kubernetes API valueFrom can also refer to values in the pod labels/annotations for specific Kubernetes cluster information such as namespace, application labels, and service annotations.

  4. Allow users to reuse stock Tekton catalogs on different cloud environment by setting up a cloud specific global container spec.

  5. User should be able to configure environment variable which needs to be provided to all workloads like for proxy settings, Users need to set environment variables like HTTP_PROXY, HTTPS_PROXY and NO_PROXY on all steps/containers.

Requirements

  1. Need to define a spec field to provide environment variables at PipelineRun and TaskRun level.
  2. Need to provide a way to define a list of global environment variables.
  3. Need to provide a way to define a list of cluster-wide forbidden environment variables. When users define environment variables in the Taskrun and Pipelinerun spec, check the list of forbidden environment variables and throw out a validation error if any of the environment variables is forbidden.
  4. Since there are many places like listed below where user can define the environment variables, there needs to be a precedence order for different places.
    • Global Level Forbidden Environment Variables
    • Global Level Default Environment Variables in Tekton Default Pod Template
    • PipelineRun Level Environment Variables in PipelineRun Pod Template
    • TaskRun Level Environment Variables in TaskRun Pod Template
    • Task Level Environment Variables in Task Step Template
    • Step Level Environment Variables in Step Container Spec

Proposal

  1. Environment variables can be defined in a single place on a PipelineRun and TaskRun level. Values can be specified as literals or through references, e.g.:
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: mypipelinerun
spec:
  podTemplate:
    env:
    - name: TOKEN_PATH
      value: /opt/user-token
    - name: TKN_PIPELINE_RUN
      valueFrom:
        fieldRef:
          fieldPath: metadata.labels['tekton.dev/pipelineRun']
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: mytaskrun
  namespace: default
spec:
  taskRef:
    name: mytask
  podTemplate:
    env:
    - name: TOKEN_PATH
      value: /opt/user-token
    - name: TKN_TASKRUN_RUN
      valueFrom:
        fieldRef:
          fieldPath: metadata.labels['tekton.dev/taskRun']
  1. Environment variables defined in steps can be easily overwritten by the ones from a TaskRun, e.g.:
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: mytask
  namespace: default
spec:
  steps:
    - name: echo-msg
      image: ubuntu
      command: ["bash", "-c"]
      args: ["echo $MSG"]
      envs:
      - name: "MSG"
        value: "Default message"
---
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: mytaskrun
  namespace: default
spec:
  taskRef:
    name: mytask
  podTemplate:
    envs:
      - name: "MSG"
        value: "Overwritten message"
  1. Environment variables defined in steps can be easily overwritten by the ones from a PipelineRun, e.g.:
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: mytask
  namespace: default
spec:
  steps:
    - name: echo-msg
      image: ubuntu
      command: ["bash", "-c"]
      args: ["echo $MSG $SECRET_PASSWORD $NAMESPACE"]
      envs:
      - name: "MSG"
        value: "Default message"
      - name: "SECRET_PASSWORD"
        value: "Default secret password"
      - name: "NAMESPACE"
        value: "tekton-pipelines"
---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: one-task-pipeline-run
  namespace: default
spec:
  pipelineSpec:
    tasks:
      - name: mytaskrun
        taskRef:
          name: mytask
  podTemplate:
    envs:
      - name: "MSG"
        value: "Overwritten message"
  1. Promote reusable tasks, users needs not to define different tasks just for the change of env value.
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: mytask
  namespace: default
spec:
  steps:
    - name: echo-msg
      image: ubuntu
      command: ["bash", "-c"]
      args: ["echo $MSG $SECRET_PASSWORD $NAMESPACE"]
      envs:
        - name: "MSG"
          value: "Default message"
        - name: "SECRET_PASSWORD"
          value: "Default secret password"
        - name: "NAMESPACE"
          value: "tekton-pipelines"
---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: three-task-pipeline-run
  namespace: default
spec:
  pipelineSpec:
    tasks:
      - name: mytaskrun
        taskRef:
          name: mytask
      - name: mytaskrun2
        taskRef:
          name: mytask
        runAfter:
          - mytaskrun
      - name: mytaskrun3
        taskRef:
          name: mytask
        runAfter:
          - mytaskrun2
  podTemplate:
    envs:
      - name: "MSG"
        valueFrom:
          fieldRef:
            fieldPath: metadata.labels['messages']
      - name: "SECRET_PASSWORD"
        valueFrom:
          secretKeyRef:
            name: mysecret
            key: password
            optional: false
      - name: "NAMESPACE"
        valueFrom:
          fieldRef:
            fieldPath: metadata.namespace

Without the ENV in podTemplate, every new PipelineRun above will need to create a new Task resource using stepTemplate to run the same examples if there is some change in env value. e.g.:

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: mytask
  namespace: default
spec:
  stepTemplate:
    envs:
    - name: "MSG"
      value: "Overwritten message"
  steps:
    - name: echo-msg
      image: ubuntu
      command: ["bash", "-c"]
      args: ["echo $MSG $SECRET_PASSWORD $NAMESPACE"]
      envs:
      - name: "MSG"
        value: "Default message"
      - name: "SECRET_PASSWORD"
        value: "Default secret password"
      - name: "NAMESPACE"
        value: "tekton-pipelines"
---
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: mytask2
  namespace: default
spec:
  stepTemplate:
    envs:
      - name: "MSG"
        valueFrom:
          fieldRef:
            fieldPath: metadata.labels['messages']
      - name: "SECRET_PASSWORD"
        valueFrom:
          secretKeyRef:
            name: mysecret
            key: password
            optional: false
      - name: "NAMESPACE"
        valueFrom:
          fieldRef:
            fieldPath: metadata.namespace
  steps:
    - name: echo-msg
      image: ubuntu
      command: ["bash", "-c"]
      args: ["echo $MSG $SECRET_PASSWORD $NAMESPACE"]
      envs:
      - name: "MSG"
        value: "Default message"
      - name: "SECRET_PASSWORD"
        value: "Default secret password"
      - name: "NAMESPACE"
        value: "tekton-pipelines"
---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: one-task-pipeline-run
  namespace: default
spec:
  pipelineSpec:
    tasks:
      - name: mytaskrun
        taskRef:
          name: mytask
---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: three-task-pipeline-run
  namespace: default
spec:
  pipelineSpec:
    tasks:
      - name: mytaskrun
        taskRef:
          name: mytask2
      - name: mytaskrun2
        taskRef:
          name: mytask2
        runAfter:
          - mytaskrun
      - name: mytaskrun3
        taskRef:
          name: mytask2
        runAfter:
          - mytaskrun2
  1. Another use case is where admin can define a list of immutable environment variables in the cluster-wide configmap. Then, user will get error if these envs get defined in podtemplate of PipelineRun or TaskRun. List of forbidden environment variables are located at the "config-defaults" configmap at the Tekton controller namespace.

Tekton configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-defaults
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/instance: default
    app.kubernetes.io/part-of: tekton-pipelines
data:
  pod-template-rules:
    forbidden-env-variables:
    - "HTTP_PROXY"

Tekton pipelinerun

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: mytask
  namespace: default
spec:
  steps:
    - name: echo-msg
      image: ubuntu
      command: ["bash", "-c"]
      args: ["echo $HTTP_PROXY"]
---
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: mytaskrun
  namespace: default
spec:
  taskRef:
    name: mytask
  podTemplate:
    envs:
      - name: "HTTP_PROXY"
        value: "8080"

The above pipeline will return "HTTP_PROXY" is not a valid environment variable to define in the podTemplate.

  1. User can also define a list of envs as part of global default pod template. All the tasks will have those envs set on steps.

Tekton configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-defaults
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/instance: default
    app.kubernetes.io/part-of: tekton-pipelines
data:
  default-pod-template:
    envs:
    - name: "MSG"
      value: "Default message"

Tekton pipelinerun

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: mytask
  namespace: default
spec:
  steps:
    - name: echo-msg
      image: ubuntu
      command: ["bash", "-c"]
      args: ["echo $MSG"]
---
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: mytaskrun
  namespace: default
spec:
  taskRef:
    name: mytask

The default "MSG" environment variable will be available in step since it's set as part of the global default pod template.

Notes/Caveats (optional)

  1. Forbidden Environment Variables are for pod template of TaskRun and PipelineRun, are they also valid for global default pod Template? How will it be avoided then? How user will get the error for that?

  2. If forbidden environment variables are specified, should they be avoided at Task's stepTemplate field or Step's Env field too ?

  3. Will there be option to pass different values of the same env, for different task in a pipeline? There can be a case that the same env requires different value in different tasks of a pipeline, and if we put it at pipelineRun
    spec level, then we are overwriting in all tasks.

Risks and Mitigations

In case of some environment variables are not allowed to change, it can have a feature flag to opt-in with this new feature. Similar to the alpha API feature flag, we can let the validation webhook fail with an error message when the feature flag is disabled. The default behavior can change after 9 months of the Tekton release cycles.

User Experience (optional)

Performance (optional)

Design Details

Bring the task stepTemplate spec to the taskRuns and pipelineRuns. Similar to stepTemplate, pipelineRun podTemplate can overwrite the taskRun and taskRun podTemplate can overwrite stepTemplate.

PipelineRun and TaskRun API Changes

In the podTemplate struct, add the env variable field like:

type Template struct {
	...
	
	// List of environment variables that can be provided to the containers belonging to the pod.
	// +optional
	Envs []corev1.EnvVar `json:"envs,omitempty" patchStrategy:"merge,retainKeys" patchMergeKey:"name" protobuf:"bytes,1,rep,name=envs"`
	
	...
}

PipelineRun spec already have podTemplate field, so new Envs field will be available here

type PipelineRunSpec struct {
	....
	// PodTemplate holds pod specific configuration
	PodTemplate *PodTemplate `json:"podTemplate,omitempty"`
    ....
}

Also, TaskRun spec have podTemplate field, so Envs will be available here

type TaskRunSpec struct {
	...
	
	// PodTemplate holds pod specific configuration
	PodTemplate *PodTemplate `json:"podTemplate,omitempty"`
	
	...
}

Global Default Environment Variables

Global level default can be provided at the config-default configmap under the default-pod-template field and the same podTemplate field is getting used to read the default values.

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-defaults
  namespace: tekton-pipelines
data:
  default-pod-template:
    envs:
    - 
    -
    -
    ...

Precedence Order

We need to have the controller logic to act based on the envs available at different places. Controller should read all and merge them till last stage which is converting them to container. The order of Precedence will look like this with first being the higher.

a. Environment Variables in PipelineRun Pod Template or TaskRun Pod Template (Same priority as both are different places)
c. Global Level Default Environment Variables in Tekton Default Pod Template
d. Task Level Environment Variables in Task Step Template
e. Step Level Environment Variables in Step Container Spec

If some default is configured and user is providing as the time of PipelineRun and TaskRun, then it should overwrite the default provided at global level.

Forbidden Environment Variables

There should be way to define the forbidden environment variables, which should not be overwritten by PipelineRun or TaskRun. If done so, webhook will throw an error that these are forbidden envs.

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-defaults
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/instance: default
    app.kubernetes.io/part-of: tekton-pipelines
data:
  pod-template-rules:
    forbidden-env-variables : 
    - "HTTP_PROXY"

Test Plan

Design Evaluation

Drawbacks

Alternatives

Define with a Top-level environment variable field. This new top-level field will be under the pipelinerun/taskrun spec level. Since the requirements also need to support Kubernetes value references such as secret, configmap, and Kubernetes downstream API, the type for this new spec will be an array of Kubernetes container V1 environment variable types.

apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: mytaskrun
  namespace: default
spec:
  taskRef:
    name: mytask
  envs:
    - name: "HTTP_PROXY"
      value: "8080"
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: one-task-pipeline-run
  namespace: default
spec:
  pipelineSpec:
    tasks:
      - name: mytaskrun
        taskRef:
          name: mytask
  envs:
    - name: "MSG"
      value: "Overwritten message"

Infrastructure Needed (optional)

Upgrade & Migration Strategy (optional)

No impact on existing features. No breaking APIs. But if users configures global default environment variable, then task env may get overridden and env have different values.

Implementation Pull request(s)

  1. Previously open: tektoncd/pipeline#3566
  2. tektoncd/pipeline#5699

References (optional)

Previously open

  1. tektoncd/pipeline#1606
  2. tektoncd/pipeline#3090