Expose pipeline run metadata to Tasks and Conditions

[dibyom]

Condition containers need access to metadata about the pipeline run. The metadata will be in the form of a pr-metadata.json file that is present in the workspace directory.

Part of implementation of #27

ref: #1137

[dibyom]

/assign

[dibyom]

Created a design proposal here: https://docs.google.com/document/d/1eJBb2VHuSTbrIWfPE3TGdRRfdbJsRCOBIqfTwx9M3H8/edit#heading=h.2f7asw3l4lo9

(to view this join the tekton-dev@ mailing list: https://github.com/tektoncd/community/blob/master/contact.md#mailing-list )

[vdemeester]

/kind feature
/area api

[bobcatfish]

@dibyom are you still working on this or can we throw it back in the backlog?

[dibyom]

@dibyom are you still working on this or can we throw it back in the backlog?

Thanks for the reminder. Moved to backlog -- here's why:

1. There is a workaround that @akihikokuroda mentioned in the design doc: Expose pipelinerun status to taskrun #1016 (comment)

2. The injected pipelinerun status is likely to be stale if there are parallel taskruns -- to counter the user would have to resort to the workaound above

3. Probably not going to be useful until we have Allowing for actions to be taken in failure scenarios #1376 as well.

I think we should take another look at this post #1376

[pritidesai]

This feature is essential for finally, final tasks are executed after all Pipeline Tasks are finished executing with success or after a single Pipeline Task fails. The scope of this design goes beyond just conditional containers, and applies to all final task containers and possibly all pipeline task containers.

Design Doc

@dibyom I would like to collaborate on the design if I can, planning on working on it after core finally functionality is merged (PR #2650 and PR #2661). Looking at the design right now 👀

Assigning it to myself as well, please let me know if there is any issue 🙏

/assign

[dibyom]

Excellent!! 😃
I had a old proposal though I am not sure how much of that is still relevant especially with finally containers! Feel free to suggest something different it that makes more sense!

[bobcatfish]

I think @dibyom 's proposal to provide this info via json file is a great first step toward what we need for both conditions + finally. We can always follow that up with more power around variable interpolation if needed.

[pritidesai]

@popcor255 do you have any particular use case or story you would like to be supported with this feature request?

[popcor255]

@pritidesai Yes, as a user I would like to log my pipelines with something with logDNA/ELK. However, this work around looks like I should be able to get the name of the pipelinerun I am under. This should let me use a step with kubctl to get the params. 😄

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: fetch-my-pipelinerunner
spec:
  steps:
  - name: kubeconfig
    image: bitnami/kubectl
    script: |
      kubectl get pipelineruns $PIPELINERUN -o json
    env:
    - name: PIPELINERUN
      valueFrom:
        fieldRef:
          fieldPath: metadata.labels['tekton.dev/pipelineRun']

[riceluxs1t]

@jerop ty for pointing this out. Will this be included in the next minor version release 0.20.0? Or could this be part of 0.19.x?

[pritidesai]

@riceluxs1t this will not be part of 0.19.x, we avoid introducing features in a patch release. It can be part of 0.20 depending on the reviews. I have added it to the 0.20 milestone for now.

[pritidesai]

Hi. We also have a use case where a Tekton pipeline consists of task A -> task B -> finally task C. The finally task C wants to perform a different logic if either task A or task B fails. In our case, the pipeline topology is fixed and the finally task C can enumerate over the status of its previous two tasks. However, like others suggested, aggregate information would be useful and save the task C the enumeration job.

I mainly wonder if this feature is being currently worked on and if so, how I can track its progress

@riceluxs1t aggregate status is still not implemented. PR #3390 has implementation of accessing individual task (task A and task B in your use case and have finally task C enumerate on task A and task B).

I will have a TEP this week to propose enumerated status of all DAG tasks.

[riceluxs1t]

@pritidesai an ability for a finally task to access the status of individual tasks is sufficient for my use case but thank you for giving me this kind update on the progress

[jrcast]

@pritidesai is there a way to get the aggregate of all tasks? Based on #3390 it seems we have to spell out each task.

      params:
        - name: task1Status
          value: "$(tasks.task1.status)"

[pritidesai]

yes @jrcast it is spelling out each task for now 🙃 . I am in process of proposing aggregate status. Please expect TEP followed by implementation PR.

[pritidesai]

/reopen

PR #3390 introduced a part of this feature, we are still working on it.

[pritidesai]

@riceluxs1t can you please review the proposal tektoncd/community#371 and let me know if it matches your expectations? 🙏

[pritidesai]

Reopening again 😜 since we do not expose the entire metadata yet, so far we have access to execution status of individual pipelineTask from the tasks section and an aggregate status of the entire tasks section, documented here.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.