TEP-0063: Workspace Dependencies

[jerop]

In #413, we added the problem statement for TEP-0063: Workspace Dependencies that discusses the need to give users a way to specify resource dependencies based on Workspaces. That will ensure that failure and skipping strategies for common CI/CD use cases work and users don't get unexpected failures.

In this change, we add the proposal and discuss the alternatives for solving that problem. To enable users to specify resource dependencies based on Workspaces, we will provide a field - useAfter - that can be used to specify that a given Task should use a specific Workspace after another Task has already used it. It will be used to construct the Directed Acyclic Graph that represents the Pipeline to enforce the execution order.

/kind tep

[tekton-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign jerop
You can assign the PR to them by writing /assign @jerop in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• teps/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ghost]

👍 this looks great from my pov. thanks @jerop !

/lgtm

[afrittoli]

/cc @pritidesai

[afrittoli]

/assign @pritidesai

[nikhil-thomas]

/assign @nikhil-thomas

[pritidesai]

@pritidesai needs review

[tekton-robot]

New changes are detected. LGTM label has been removed.

[pritidesai]

Today, if the WhenExpressions in manual-approval evaluate to False, then manual-approval, build-image,
deploy-image and slack-msg are all skipped. In TEP-0059 Skip Guarded Task Only,
we'll provide the flexibility to unblock the execution of build-image, deploy-image and slack-msg when
manual-approval is skipped. This would allow the user to reuse the Pipeline in both scenarios (merging and not merging).

If the approver resource is a Result, we can identify it as a resource dependency and skip slack-msg. However, if the
approver resource is passed through a Workspace, we can't identify that as a resource dependency. So we'll execute
slack-msg which would fail because it can't resolve missing resources. When slack-msg fails, the whole Pipeline
would fail.

Sorry @jerop for going back to the use cases added in the initial PR.

I think these two statements contradict each other. The first para says TEP-0059 unblocks the execution of the slack-msg when manual-approval is skipped. slack-msg is a resource dependent through a task result on manual-approval. My understanding is at least the slack-msg is scheduled and attempted, the result references are resolved and then marked as skipped if the result resolution is not successful without creating a taskRun.

In the second para, we are saying the resource dependency is discovered and will skip slack-msg.

[afrittoli]

Thank you for this @jerop !

I think I understand the problem, however I don't see what a pipeline author could express through the proposed syntax that cannot already be expressed with the existing runAfter?

The reason for this is that the dependency still has to be expressed as a list of tasks as opposed to a list of resources produced by these tasks.

We lack in Tekton today the ability to express such resources though.
An approximation could be folders or files in a workspace?

[bobcatfish]

The reason for this is that the dependency still has to be expressed as a list of tasks as opposed to a list of resources produced by these tasks.

I'm not totally following @afrittoli - I think the proposed syntax would contain both? e.g.

      workspaces:
        - name: input
          workspace: shared-data
          useAfter: 
          - task1

This would be saying that the resource shared-data should be "used after" task1 which implies that this task expects something task1 to do something to shared-data (which could be mutating it, or even could be something like verifying the content).

With today's runAfter syntax, users can express ordering but nothing more - i.e. they can't express when a task depends on another task doing something with a workspace (i.e. a "resource dependency" vs an "ordering dependency")

[vdemeester]

      workspaces:
        - name: input
          workspace: shared-data
          useAfter: 
          - task1

This would be saying that the resource shared-data should be "used after" task1 which implies that this task expects something task1 to do something to shared-data (which could be mutating it, or even could be something like verifying the content).

With today's runAfter syntax, users can express ordering but nothing more - i.e. they can't express when a task depends on another task doing something with a workspace (i.e. a "resource dependency" vs an "ordering dependency")

How "important" is that to express when a task depends on another task doing something with a workspace ? What I mean is, as proposed, nothing ensure/validates that the previous task has done something with the workspace, very similar to runAfter. What could be done is validating that the workspace used in task B is also bound to task A (but I don't see it described in the TEP). The Workspace TEP was a bit more involved n that.

[vdemeester]

Could we show/describe what those workarounds are and how they reduce the re-usability ?