Optimization of copying output resources to PVC(or intermediate storage location)

[shashwathi]

Expected Behavior

Copy resources onto PVC(or intermediate storage location) only if resource is being used further in pipelinerun

Actual Behavior

TaskRun copies are output resources to PVC irrespective of it being used further in taskrun in pipelinerun or not.

Steps to Reproduce the Problem

1. Create pipelinerun with a task
2. Check the pod spec to see 2 steps (in the end) for create a directory in PVC and copying contents

Additional Info

Could consider using DAG functionality to determine whether a resource in output is used in further tasks and make the right decision

[afrittoli]

Output/logs will still be on the Task own PVC, right, so this would only change whether they are copied on the pipeline PVC, right?

[bobcatfish]

hm interesting, I wonder how we can make the TaskRun aware of this without making it too aware of or coupled to Pipeline runs 🤔

@shashwathi is this mostly to save time? or save storage space? something else?

Output/logs will still be on the Task own PVC, right, so this would only change whether they are copied on the pipeline PVC, right?

That's right @afrittoli but just note that for #387 it's pretty likely we'll end up getting rid of that log PVC!

[shashwathi]

is this mostly to save time?

Yes mostly to avoid adding steps that are not adding value. It also saves execution time of taskrun

Logs are not in the scope of this issue. It is about intelligently handling output resources instead of copying (in dumb manner?)

[cccfeng]

Hi everyone, I totally agree with what @shashwathi say.

In the following user scenarios, whether a pipeline containing only one task is defined or an output resource in a task is not referenced by subsequent tasks, Tekton may not need to copy output resources into PVC or remote bucket. This logic is very clever, but it may not be very appropriate as the default logic.

I think it's better to parameterize this action and let the user configure it visually, rather than deduce the logic of copying output resource from DAG. Users will be aware of this, just as for a CI task run-out artifact, users can specify that it is only copied to a remote bucket for storage, or whether it is copied to a PVC pulled up by pipeline for subsequent ls/cat actions or other purposes.

Proposal

Whether it can be or not to extend pipeline.Spec.tasks[n].resources.outputs[n] (PipelineTaskOutputResource) to have such as CustomUploadOnly bool (or something more expressive) attribute

• CustomUploadOnly with default value false , output resource will be copied into pvc/bucket that retain the logic before
• When set customUploadOnly to true , output resource will not be copied into pvc/bucket by system. Users can use GCS or any other extended pipeline resources, such as OSS to store artifact

Yaml

kind: Pipeline
metadata:
  name: demo-pipeline
spec:
  resources:
  - name: source-repo
    type: git
  - name: ci-artifact
    type: storage
  tasks:
  - name: xxxx-ci-pt
    taskRef:
      name: xxx-ci-task
    resources:
      inputs:
      - name: workspace
        resource: source-repo
      outputs:
      - name: builtImage
        resource: web-image
        customUploadOnly: true

I am looking forward to your reply , thanks.

[ghost]

I believe this is done now. A PVC is only created and copied to if an input links from an output. Feel free to reopen if there's more to do here!