Should the compiler generate Pipeline or PipelineRun

[ckadner]

This is a general discussion, not a work item.

Conceptually a Pipeline is a static specification orchestrating how a set of Tasks should be executed. A PipelineRun is an instantiation of a pipeline. A pipeline is a re-usable artifact that can be "run" repeatedly, a pipeline-run is one-and-done.

Currently the KFP-Tekton compiler generates a YAML with (multiple) Tasks(s) and one Pipeline document. There are however certain features supported by the KFP DSL which are not easily transferred into an equivalent feature in Tekton Pipelines, but instead require the concept of a PipelineRun, like mounting PVC volume to a pipeline workspace or setting a timeout at the pipeline level.

Linked issues

• Pipeline level timeout Pipeline level timeout #56
• Passing parameters between tasks #41
• Volume mounts add support for dsl.volumeOps #51
• Generate PipelineRun with embedded pipelineSpec and taskSpec Generate PipelineRun with embedded pipelineSpec and taskSpec #143

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
question	0.84

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[Tomcli]

For comparison, every workflow in Argo is like a PipelineRun. Therefore, when we upload a pipeline in kfp, it's treated as a template and stored in object storage instead of applying the template to Kubernetes.

[ckadner]

For comparison, every workflow in Argo is like a PipelineRun. Therefore, when we upload a pipeline in kfp, it's treated as a template and stored in object storage instead of applying the template to Kubernetes.

True. But for Kubeflow Pipelines with Argo there is the KFP UI which allows users to interact with pipelines and provide parameters and see logs, etc. For Tekton we have to rely on kubectl and tkn. If we generate PipelineRuns then a user cannot provide parameters at run-time, rather parameters, volume mounts, PVC are locked in at compilation time.

[ckadner]

@afrittoli @animeshsingh @fenglixa @vincent-pli

[Tomcli]

tektoncd/cli#786

[ckadner]

we could also introduce a --generate-pipelinerun flag for the dsl-compile-tekton CLI

[Tomcli]

I will add a flag for development purpose and we should push the cli team to support workspaces.

[fenglixa]

Great, I think we should support generate-pipelinerun feature, it's convenient for user to run the whole workflow directly, but not just leave several templates there, also, pipelinerun has several fields which could be defined in there, including pipeline timeout.

[Tomcli]

generating pipelinerun might also be useful for adding pod-template such as node selector and affinity features https://github.com/tektoncd/pipeline/blob/master/docs/pipelineruns.md#pod-template

[fenglixa]

Personally, I think we should generate tekton pipelinerun by default to align with argo workflow but not pipeline.
We may need to list pros and cons for each optionals and adopt one of them, convenient for further API and UI integration

[animeshsingh]

Given more preference towards pipelinerun, lets go with that

[vincent-pli]

Pipelinerun with Pipeline definition embed:

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: pipelinerun-with-taskspec-to-echo-good-morning
spec:
  pipelineSpec:
    tasks:
      - name: echo-good-morning
        taskSpec:
          steps:
            - name: echo
              image: ubuntu
              script: |
                #!/usr/bin/env bash
                echo "Good Morning!"

[ckadner]

PipelineRun with Pipeline definition embed

I think @Tomcli 's approach (PR #62) with generating an optional PipelineRun in addition to the existing Pipeline with Tasks will be the best compromise and allows to keep existing code intact. At least for the time being.

[afrittoli]

PipelineRun with Pipeline definition embed

I think @Tomcli 's approach (PR #62) with generating an optional PipelineRun in addition to the existing Pipeline with Tasks will be the best compromise and allows to keep existing code intact. At least for the time being.

Eventually you could have a flag that let you decide whether to embed the pipeline in the pipelinerun or not.

[afrittoli]

Whether you want to use Pipeline or PipelineRun depends on what kind of experience you want to give to end-users.

Tekton separates static definitions from runtimes and it provides a component called “triggers” that allow you to create runtime objects with several presets. Today “triggers” only respond to HTTPs requests, but I can imagine a future were it will be possible to run “triggers” in different ways.
It seems to be that that separation between static definition and runtime is not so clear in the python sdk, so after reading all the discussion it feels like you may have to go with generating PipelineRuns. If you still need to support changing some of the parameters at run time, you could support that using kustomize overlay.

[ckadner]

/kind discussion

[ckadner]

/remove-kind question

[ckadner]

Related issues:

• Generate PipelineRun with embedded pipelineSpec and taskSpec Generate PipelineRun with embedded pipelineSpec and taskSpec #143 PipelineRun with embedded pipelineSpec and taskSpec #166
• Unsupported features for embedded pipelineSpec, taskSpec? tektoncd/pipeline #2767

[Udiknedormin]

Shouldn't this be closed, considering issue #143 has been closed with PR #166?

[Tomcli]

Thanks @Udiknedormin
/close

[k8s-ci-robot]

@Tomcli: Closing this issue.

In response to this:

Thanks @Udiknedormin
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.