Add volumeClaimTemplate as a Workspace volume source #2326

jlpettersson · 2020-04-04T18:29:21Z

Changes

An existing PersistentVolumeClaim can currently be used as a Workspace
volume source. There is two ways of using an existing PVC as volume:

Reuse an existing PVC
Create a new PVC before each PipelineRun.

There is disadvantages by reusing the same PVC for every PipelineRun:

You need to clean the PVC at the end of the Pipeline
All Tasks using the workspace will be scheduled to the node where
the PV is bound
Concurrent PipelineRuns may interfere, an artifact or file from one
PipelineRun may slip in to or be used in another PipelineRun, with
very few audit tracks.

There is also disadvantages by manually creating a new PVC before each PipelineRun:

This can not (easily) be done declaratively
This is hard to do programmatically, because it is hard to know when
to delete the PVC. The PipelineRun can not be set as OwnerReference since
the PVC must be created first

This commit adds 'volumeClaimTemplate' as a volume source for workspaces. This
has several advantages:

The syntax is used in k8s StatefulSet and other k8s projects so it is
familiar in the kubernetes ecosystem
It is possible to declaratively declare that a PVC should be created for each
PipelineRun, e.g. from a TriggerTemplate.
The user can choose storageClass (or omit to get the cluster default) to e.g.
get a faster SSD volume, or to get a volume compatible with e.g. Windows.
The user can adapt the size to the job, e.g. use 5Gi for apps that contains
machine learning models, or 1Gi for microservice apps. It can be changed on
demand in a configuration that lives in the users namespace e.g. in a
TriggerTemplate.
The size affects the storage quota that is set on the namespace and it may affect
billing and cost depending on the cluster environment.
The PipelineRun or TaskRun with the template is created first, and is used
as OwnerReference on the PVC. That means that the PVC will have the same lifecycle
as the PipelineRun.

How it works

When using a volumeClaimTemplate workspace on a PipelineRun:

Before the first TaskRun is created; a PVC from the template is created - with the PipelineRun as OwnerReference.
The volumeClaimTemplate workspace is translated to a PersistentVolumeClaim workspace when creating TaskRuns.
Whenever the PipelineRun is deleted, the PVC is also deleted.

When using volumeClaimTemplate workspace on a TaskRun not initiated from a PipelineRun:

Before the Pod is created; a PVC from the template is created - with the TaskRun as OwnerReference.
The volumeClaimTemplate workspace is translated to a PersistentVolumeClaim workspace when creating the Pod.
Whenever the TaskRun is deleted, the PVC is also deleted.

Example usage:

apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
  name: pipeline-template
spec:
  resourcetemplates:
  - apiVersion: tekton.dev/v1beta1
    kind: PipelineRun
    metadata:
      generateName: simple-pipeline-run-
    spec:
      pipelineRef:
        name: simple-pipeline
      workspaces:
      - name: ws
        volumeClaimTemplate:
          metadata:
            name: ws-pvc
          spec:
            accessModes: ["ReadWriteOnce"]
            resources:
               requests:
                  storage: 1Gi

Related to #1986

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

Includes tests (if functionality changed/added)
Includes docs (if user facing)
Commit messages follow commit message best practices

Reviewer Notes

If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS, and they must first be added in a backwards compatible way.

Release Notes

Add volumeClaimTemplate as a Workspace volume source

When using volumeClaimTemplate as volume source for a PipelineRun, the
tekton-controller will create a PersistentVolumeClaim using the template, 
and be seen as a PersistentVolumeClaim for all TaskRuns in the Pipeline. 
The PersistentVolumeClaim will be deleted when the PipelineRun is deleted.

tekton-robot · 2020-04-04T18:29:36Z

Hi @jlpettersson. Thanks for your PR.

I'm waiting for a tektoncd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ghost · 2020-04-05T11:41:09Z

/ok-to-test

/hold

Thanks very much for the PR, it's exciting to see this come together! I'm adding the hold just so that we can raise this with the Working Group on Wednesday and share it more widely for any feedback.

tekton-robot · 2020-04-05T11:44:01Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/pipelinerun_types.go	90.9%	88.5%	-2.4
pkg/apis/pipeline/v1alpha1/taskrun_types.go	90.5%	84.6%	-5.9
pkg/reconciler/volumeclaim/pvchandler.go	Do not exist	87.0%
test/builder/pipeline.go	87.1%	86.1%	-1.1
test/builder/task.go	68.2%	67.7%	-0.5

ghost · 2020-04-06T11:52:20Z

There were a few lint errors, which is why pull-tekton-pipeline-build-tests failed:

I0405 12:36:03.429] ===============================
I0405 12:36:03.429] ==== RUNNING GOLANGCI-LINT ====
I0405 12:36:03.429] ===============================
I0405 12:37:09.781] pkg/reconciler/volumeclaim/pvchandler.go:4: File is not `goimports`-ed (goimports)
I0405 12:37:09.781] 	"fmt"
I0405 12:37:09.782] pkg/reconciler/volumeclaim/pvchandler_test.go:4: File is not `goimports`-ed (goimports)
I0405 12:37:09.782] 	"fmt"
I0405 12:37:09.782] pkg/reconciler/volumeclaim/pvchandler.go:54:59: S1019: should use make(map[string]*corev1.PersistentVolumeClaim) instead (gosimple)
I0405 12:37:09.782] 	claims := make(map[string]*corev1.PersistentVolumeClaim, 0)

The integration tests look like they timed out. Re-running to check if this reproduces.

/test pull-tekton-pipeline-integration-tests

tekton-robot · 2020-04-06T12:35:04Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/pipelinerun_types.go	90.9%	88.5%	-2.4
pkg/apis/pipeline/v1alpha1/taskrun_types.go	90.5%	84.6%	-5.9
pkg/reconciler/volumeclaim/pvchandler.go	Do not exist	87.0%
test/builder/pipeline.go	87.1%	86.1%	-1.1
test/builder/task.go	68.2%	67.7%	-0.5

jlpettersson · 2020-04-06T14:01:24Z

/test pull-tekton-pipeline-build-tests

tekton-robot · 2020-04-07T11:52:32Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/pipelinerun_types.go	90.9%	88.5%	-2.4
pkg/apis/pipeline/v1alpha1/taskrun_types.go	90.5%	84.6%	-5.9
pkg/reconciler/volumeclaim/pvchandler.go	Do not exist	87.0%
test/builder/pipeline.go	84.0%	83.0%	-1.0
test/builder/task.go	68.2%	67.7%	-0.5

ghost

Thank you very much for adding this! I've added some comments after a quick read-through. I plan to check out this PR locally and give it a try over the next few days as well.

It would be useful to include some examples in the examples/v1beta1/taskruns and examples/v1beta1/pipelineruns directories. These YAML files will be executed as part of our E2E suite and are a good way to regression-test behaviour. And it functions as useful documentation too!

It would also be great to introduce explicit E2E testing to our test directory that checks scenarios like:

what happens when multiple tasks in a single pipeline share a claim + workspace binding name + owner name. (But only if this is a situation that can actually happen ofcourse)
how a PipelineRun / TaskRun fails when a volume claim template cannot be created.

pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go

pkg/reconciler/volumeclaim/pvchandler.go

ghost · 2020-04-07T13:05:16Z

pkg/reconciler/volumeclaim/pvchandler.go

+
+// getPersistentVolumeClaimName gets the name of PersistentVolumeClaim for a Workspace and PipelineRun or TaskRun. claim
+// must be a PersistentVolumeClaim from set's VolumeClaims template.
+func GetPersistentVolumeClaimName(claim *corev1.PersistentVolumeClaim, wb v1alpha1.WorkspaceBinding, owner metav1.OwnerReference) string {


Is this guaranteed to be unique? I'm trying to figure out the possible combinations and I have a hunch that there could be two Tasks in a Pipeline that share a PVC Claim Name, WorkspaceBinding Name, and PipelineRun (owner) Name. In this case would there be potential for collision? And is that problematic?

Thanks for reviewing! There are two cases. I tried to explain them in my PR description under "How it works". When the volumeClaimTemplate is declared for a PipelineRun (owner) - it will be seen as a "PersistentVolumeClaim"-workspace for all TaskRuns in the Pipeline. The function taskWorkspaceByWorkspaceVolumeSource() in pipelinerun.go applies the template and pass it as PersistentVolumeClaim to the TaskRuns - this happens after the PVC is created from the template - after the IsBeforeFirstTaskRun()-check. I can explain this more in person if you want.

PipelineRun has a unique name, and will be owner.Name here. There can also be multiple workspaces per PipelineRun but they must have unique names. Hence the PVC get an unique name here.

tekton-robot · 2020-04-07T15:15:17Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/pipelinerun_types.go	90.9%	88.5%	-2.4
pkg/apis/pipeline/v1alpha1/taskrun_types.go	90.5%	84.6%	-5.9
pkg/reconciler/volumeclaim/pvchandler.go	Do not exist	87.0%
test/builder/pipeline.go	84.0%	83.0%	-1.0
test/builder/task.go	68.2%	67.7%	-0.5

jlpettersson · 2020-04-07T15:45:09Z

/retest

ghost · 2020-04-07T16:25:08Z

I think the integration is failing because of #2337 at the moment. This is something I'd like to fix for the next point release, which is supposed to happen tomorrow. Will try to get this resolved asap.

ghost · 2020-04-08T18:07:18Z

/hold cancel

Thanks for presenting to the WG! Am going to work on fixing these integration tests and then will get back to this review.

tekton-robot · 2020-04-09T00:47:08Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/pipelinerun_types.go	90.9%	88.5%	-2.4
pkg/apis/pipeline/v1alpha1/taskrun_types.go	90.5%	84.6%	-5.9
pkg/reconciler/volumeclaim/pvchandler.go	Do not exist	87.0%
test/builder/pipeline.go	84.0%	83.0%	-1.0
test/builder/task.go	68.2%	67.7%	-0.5

vdemeester

Looking good, only few small comments (on naming)

vdemeester · 2020-04-09T13:25:29Z

docs/workspaces.md

 `PersistentVolumeClaim` volumes are a good choice for sharing data among `Tasks` within a `Pipeline`.

+#### `volumeClaimTemplate`


Any reason to name it volumeClaimTemplate and not persistentVolumeClaim ?
(Mainy reason for this question is if we should be consistent with the persistentVolumeClaim field)

I borrowed the whole syntax and idea with volumeClaimTemplate from StatefulSet https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#components and wanted to be consistent with that terminology. The persistentVolumeClaim is still there for workspaces and can be used if you want to use an existing PVC.

Fair enough. I don't have a strong opinion on this, although I would tend to prefer be consistent in the naming with our own API than k8s one 😛

agree with the persistence part, makes lots of sense...

vdemeester · 2020-04-09T13:36:45Z

pkg/apis/pipeline/v1alpha1/taskrun_types.go

+
+// HasVolumeClaimTemplate returns true if TaskRun contains volumeClaimTemplates that is
+// used for creating PersistentVolumeClaims with an OwnerReference for each run
+func (tr *TaskRun) HasVolumeClaimTemplate() bool {


Shouldn't those changes be also on v1beta1 ? (even though this is the one used by the controller)

Good catch! I should add it to v1beta1 as well, yes.

I added the the changes I made on v1alpha1 to v1beta1 and also squashed my commits. Thanks for reviewing.

Btw, not sure where to ask for this, anyone can help with how would you use it in a CLI with tkn to pass the name of the template which already exists in the system (not template file)? e.g.:

I have tried this but it continuously says cannot find the PVC (NOTE: the "default-netapp-disk" exists in OpenShift as a VolumeClaimTemplate and it is the default one as well):

_tkn pipeline start pipeline-name -w name=shared-data,claimName=VolumeClaimTemplate,storageClass=default-netapp-disk,accessModes=ReadWriteOnce,storage=1Gi,volumeMode=Filesystem ....

OUTPUT:
message: >-
pod status "PodScheduled":"False"; message: "0/12 nodes are available:
12 persistentvolumeclaim "VolumeClaimTemplate" not found. preemption:
0/12 nodes are available: 12 Preemption is not helpful for scheduling."

I've also tried using 'claimName=default-netapp-disk' as well, and many other things though nothing works when it comes to sending the VCT and it's details through the tkn CLI...

This is very critical to me and I did not really found any useful info during this last week while working on it...

tekton-robot · 2020-04-09T22:15:52Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/pipelinerun_types.go	90.9%	88.5%	-2.4
pkg/apis/pipeline/v1alpha1/taskrun_types.go	90.5%	84.6%	-5.9
pkg/apis/pipeline/v1beta1/pipelinerun_types.go	87.1%	85.7%	-1.4
pkg/apis/pipeline/v1beta1/taskrun_types.go	71.4%	69.7%	-1.7
pkg/reconciler/volumeclaim/pvchandler.go	Do not exist	87.0%
test/builder/pipeline.go	84.0%	83.0%	-1.0
test/builder/task.go	68.2%	67.7%	-0.5

An existing PersistentVolumeClaim can currently be used as a Workspace volume source. There is two ways of using an existing PVC as volume: - Reuse an existing PVC - Create a new PVC before each PipelineRun. There is disadvantages by reusing the same PVC for every PipelineRun: - You need to clean the PVC at the end of the Pipeline - All Tasks using the workspace will be scheduled to the node where the PV is bound - Concurrent PipelineRuns may interfere, an artifact or file from one PipelineRun may slip in to or be used in another PipelineRun, with very few audit tracks. There is also disadvantages by creating a new PVC before each PipelineRun: - This can not (easily) be done declaratively - This is hard to do programmatically, because it is hard to know when to delete the PVC. The PipelineRun can not be set as OwnerReference since the PVC must be created first This commit adds 'volumeClaimTemplate' as a volume source for workspaces. This has several advantages: - The syntax is used in k8s StatefulSet and other k8s projects so it is familiar in the kubernetes ecosystem - It is possible to declaratively declare that a PVC should be created for each PipelineRun, e.g. from a TriggerTemplate. - The user can choose storageClass (or omit to get the cluster default) to e.g. get a faster SSD volume, or to get a volume compatible with e.g. Windows. - The user can adapt the size to the job, e.g. use 5Gi for apps that contains machine learning models, or 1Gi for microservice apps. It can be changed on demand in a configuration that lives in the users namespace e.g. in a TriggerTemplate. - The size affects the storage quota that is set on the namespace and it may affect billing and cost depending on the cluster environment. - The PipelineRun or TaskRun with the template is created first, and is used as OwnerReference on the PVC. That means that the PVC will have the same lifecycle as the PipelineRun. Related to tektoncd#1986 See also: - tektoncd#2174 - tektoncd#2218 - tektoncd/triggers#476 - tektoncd/triggers#482 - kubeflow/kfp-tekton#51

tekton-robot · 2020-04-09T22:23:13Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/pipelinerun_types.go	90.9%	88.5%	-2.4
pkg/apis/pipeline/v1alpha1/taskrun_types.go	90.5%	84.6%	-5.9
pkg/apis/pipeline/v1beta1/pipelinerun_types.go	87.1%	85.7%	-1.4
pkg/apis/pipeline/v1beta1/taskrun_types.go	71.4%	69.7%	-1.7
pkg/reconciler/volumeclaim/pvchandler.go	Do not exist	87.0%
test/builder/pipeline.go	84.0%	83.0%	-1.0
test/builder/task.go	68.2%	67.7%	-0.5

vdemeester

/lgtm
/meow

tekton-robot · 2020-04-10T10:38:23Z

@vdemeester:

In response to this:

/lgtm
/meow

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tekton-robot · 2020-04-14T15:53:42Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbwsg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sbwsg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ghost · 2020-04-14T15:54:07Z

Tried everything out locally and it worked like a charm!

ghost · 2020-04-14T16:32:01Z

/test pull-tekton-pipeline-integration-tests

bobcatfish · 2020-04-20T14:50:27Z

Thanks for adding this @jlpettersson !! I'm really excited to try it out :D

bobcatfish · 2020-04-20T19:30:06Z

@jlpettersson any chance you'd be up for adding a bit to the docs about how to use the template, e.g. what the syntax is? even just a brief example would be great! lemme know if not and ill either do it or open an issue :)

jlpettersson · 2020-04-20T21:20:53Z

@bobcatfish yes, I could improve the docs. I'll have a look tomorrow.

In addition to the example in the PR-description, there is one example in v1beta1 pipelineruns: https://github.com/tektoncd/pipeline/blob/master/examples/v1beta1/pipelineruns/workspace-from-volumeclaimtemplate.yaml

Add a short example and link to a full example of using volumeClaimTemplate as a volume source in a workspace. Requested in comment to PR tektoncd#2326 (comment) that fixes tektoncd#1986

Add a short example and link to a full example of using volumeClaimTemplate as a volume source in a workspace. Requested in comment to PR #2326 (comment) that fixes #1986

Jonas has recently become a regularly contributor. He started with adding a minor [_missing_ `omitempty`](tektoncd/pipeline#2301) and then [proposed some ideas](tektoncd/pipeline#1986 (comment)) around workspaces and PersistentVolumeClaim creation and continued to [elaborate around those ideas](tektoncd/pipeline#1986 (comment)). A sunny day a few days later, he also submitted an [extensive implementation for volumeClaimTemplate](tektoncd/pipeline#2326), corresponding to the idea discussions. A few days later submitted a [small refactoring PR](tektoncd/pipeline#2392), and he also listened to community members that [proposed changes](tektoncd/pipeline#2450) to his implementation about volumeClaimTemplates and did an [implementation for that proposal](tektoncd/pipeline#2453). A rainy day, he also wrote [technical documentation about PVCs](tektoncd/pipeline#2521) including adding an example that caused _flaky_ integration tests for the whole community during multiple days. When he understood his mistake, he submitted a [removal of the example](tektoncd/pipeline#2546) that caused flaky tests. He has also put his toe into Tekton Catalog and [contributed to the buildah task](tektoncd/pipeline#2546). This has followed, mostly with more PRs to the Pipeline project: - tektoncd/pipeline#2460 - tektoncd/pipeline#2491 - tektoncd/pipeline#2502 - tektoncd/pipeline#2506 - tektoncd/pipeline#2632 - tektoncd/pipeline#2633 - tektoncd/pipeline#2634 - tektoncd/pipeline#2636 - tektoncd/pipeline#2601 - tektoncd/pipeline#2630 Jonas is excited about the great community around Tekton and the project! He now would like to join the org.

lotosotol · 2023-08-18T23:43:12Z

@bobcatfish yes, I could improve the docs. I'll have a look tomorrow.

In addition to the example in the PR-description, there is one example in v1beta1 pipelineruns: https://github.com/tektoncd/pipeline/blob/master/examples/v1beta1/pipelineruns/workspace-from-volumeclaimtemplate.yaml

404 Page not found...

googlebot added the cla: yes Trying to make the CLA bot happy with ppl from different companies work on one commit label Apr 4, 2020

tekton-robot requested review from afrittoli and imjasonh April 4, 2020 18:29

tekton-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 4, 2020

jlpettersson mentioned this pull request Apr 4, 2020

Relax TriggerTemplate limitation of only creating Tekton resources tektoncd/triggers#482

Closed

ghost reviewed Apr 7, 2020

View reviewed changes

tekton-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 8, 2020

vdemeester reviewed Apr 9, 2020

View reviewed changes

bobcatfish mentioned this pull request Apr 9, 2020

Cannot use parameter substitution for volume size #1514

Closed

jlpettersson force-pushed the add_volumeclaimtemplate_as_workspace_volume_source branch from e7cdea6 to c7ce3da Compare April 9, 2020 22:20

vdemeester reviewed Apr 10, 2020

View reviewed changes

tekton-robot assigned vdemeester Apr 10, 2020

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Apr 10, 2020

ghost approved these changes Apr 14, 2020

View reviewed changes

tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 14, 2020

tekton-robot merged commit fe5981e into tektoncd:master Apr 14, 2020

davissp14 mentioned this pull request Apr 14, 2020

Design Concern - PipelineRuns #2372

Closed

bobcatfish mentioned this pull request Apr 20, 2020

Allow volumeClaimTemplate name to be not provided #2450

Closed

jlpettersson mentioned this pull request Apr 21, 2020

Improve documentation about volumeClaimTemplate workspaces #2460

Merged

3 tasks

afrittoli added the kind/feature Categorizes issue or PR as related to a new feature. label Apr 30, 2020

jlpettersson mentioned this pull request May 19, 2020

Add Jonas to the org tektoncd/community#106

Merged

jlpettersson mentioned this pull request Nov 25, 2020

Allow volume from volumeClaimTemplate to be used in more than one workspace #3440

Open

		`PersistentVolumeClaim` volumes are a good choice for sharing data among `Tasks` within a `Pipeline`.

		#### `volumeClaimTemplate`

Add volumeClaimTemplate as a Workspace volume source #2326

Add volumeClaimTemplate as a Workspace volume source #2326

Conversation

jlpettersson commented Apr 4, 2020 • edited Loading

Changes

Submitter Checklist

Reviewer Notes

Release Notes

tekton-robot commented Apr 4, 2020

ghost commented Apr 5, 2020

tekton-robot commented Apr 5, 2020

ghost commented Apr 6, 2020

tekton-robot commented Apr 6, 2020

jlpettersson commented Apr 6, 2020

tekton-robot commented Apr 7, 2020

ghost left a comment

Choose a reason for hiding this comment

ghost Apr 7, 2020 • edited by ghost Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tekton-robot commented Apr 7, 2020

jlpettersson commented Apr 7, 2020

ghost commented Apr 7, 2020

ghost commented Apr 8, 2020

tekton-robot commented Apr 9, 2020

vdemeester left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tekton-robot commented Apr 9, 2020

tekton-robot commented Apr 9, 2020

vdemeester left a comment

Choose a reason for hiding this comment

tekton-robot commented Apr 10, 2020

tekton-robot commented Apr 14, 2020

ghost commented Apr 14, 2020

ghost commented Apr 14, 2020

bobcatfish commented Apr 20, 2020

bobcatfish commented Apr 20, 2020

jlpettersson commented Apr 20, 2020 • edited Loading

lotosotol commented Aug 18, 2023

jlpettersson commented Apr 4, 2020 •

edited

Loading

ghost Apr 7, 2020 •

edited by ghost

Loading

jlpettersson commented Apr 20, 2020 •

edited

Loading