Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a release pipeline and update the release docs 📖 #1095

Merged
merged 1 commit into from
Aug 6, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
155 changes: 112 additions & 43 deletions tekton/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,17 @@ them manually.

## Pull Request Pipeline

The `Tasks` which are going to make up our Pull Request Pipeline are:
The pull request pipeline will use the
[`golang`](https://github.com/tektoncd/catalog/tree/master/golang)
Tasks from the
[`tektoncd/catalog`](https://github.com/tektoncd/catalog). To add them
to your cluster:

- [`ci-unit-test.yaml`](ci-unit-test.yaml) — This `Task` uses `go test` to run
the Pipeline unit tests.
- [`ci-lint.yaml`](ci-lint.yaml) — This `Task` uses
[`golangci-lint`](https://github.com/golangci/golangci-lint) to validate the
code based on common best practices.
```
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/master/golang/lint.yaml
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/master/golang/build.yaml
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/master/golang/tests.yaml
```
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😍


TODO(#922) & TODO(#860): Add the Pipeline and hook it up with Prow, for now all
we have are `Tasks` which we can invoke individually by creating
Expand All @@ -34,7 +38,19 @@ and

## Release Pipeline

The `Tasks` which make up our release `Pipeline` are:
The release pipeline uses the
[`golang`](https://github.com/tektoncd/catalog/tree/master/golang)
Tasks from the
[`tektoncd/catalog`](https://github.com/tektoncd/catalog). To add them
to your cluster:

```
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/master/golang/lint.yaml
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/master/golang/build.yaml
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/master/golang/tests.yaml
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can keep discussing this (I don't want to block merging this!) but I'm not totally sold on running all the tests as part of the release pipeline. The tests are run on the code merged with HEAD before merging, so if one of these tests fails it would mean that either 1) something we are not accounting for changed (e.g. a dependency that's being pulled in) or 2) something is flakey. I think 99.9% of the time it would be the second case, in which case the tests would just slow down the release process + result in having to run it again. I guess we would be doing this to try to catch (1)? I would rather try to catch that by running continuous tests than doing it at release time 🤔

What are your thoughts on this @vdemeester ? (Maybe this would catch something else I haven't thought of - or maybe it's just worth it for the increased confidence!)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So overall I do agree with you but there is one case that is not clear yet. I'll took 0.5.2 to explain this one — as we cherry-picked commits on the branch, we didn't do pull-request, and thus we did not run the entire test suites on it — that's mainly why I did #1088.

That's why I felt safe to run tests during the release (even if, there, it's only the unit tests 😅).

Honestly I am ok not running them but I do like the idea that if it fails because of flakyness:

  • it will make us prioritize fixing that flakyness 😝
  • it slows the release but by a tiny tiny amount of time (a matter of minutes 😝)

Copy link
Member

@afrittoli afrittoli Jul 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me the main point of doing release time testing would be doing extra testing there that is normally too time/resource consuming to run on each commit/PR - I'm not sure we have such tests (yet?) though. I guess we need to see how long the upgrade testing will take to run once it's been written.

Another way to run those kind of tests is to have a periodic (daily?) run of those tests; the issue I've seen in the past with periodic tests is that is hard to have people putting effort in keeping them green - since they don't block merging patches - but if we use such tests for pre-release it might give the extra motivation needed.

Kind of diverging from the main topic, periodic testing, if kept green, have an extra value, which is to provide a source of "clean" data to detect flaky test. Test jobs on PR may fail because the code in the PR makes them fail, however periodic tests should be 100% passing, so failures there are good data for identifying and troubleshooting flaky tests.

This said, I would not mind keeping the unit tests in there anyways.

```

The *local* `Tasks` which make up our release `Pipeline` are:

- [`ci-images.yaml`](ci-images.yaml) - This `Task` uses
[`kaniko`](https://github.com/GoogleContainerTools/kaniko) to build and
Expand All @@ -44,7 +60,14 @@ The `Tasks` which make up our release `Pipeline` are:
[`kaniko`](https://github.com/GoogleContainerTools/kaniko) to build and
publish base images, and uses
[`ko`](https://github.com/google/go-containerregistry/tree/master/cmd/ko) to
build all of the container images we release and generate the `release.yaml`
build all of the container images we release and generate the
`release.yaml`
- [`release-pipeline.yaml`](./release-pipeline.yaml) - This `Pipeline`
uses the
[`golang`](https://github.com/tektoncd/catalog/tree/master/golang)
`Task`s from the
[`tektoncd/catalog`](https://github.com/tektoncd/catalog) and
[`publish.yaml`](publish.yaml)'s `Task`.

The official releases [are performed from the `prow` cluster in the `tekton-releases`
GCP project](https://github.com/tektoncd/plumbing#prow). To release you will want to:
Expand Down Expand Up @@ -88,6 +111,10 @@ image registry instead.
- [`publish-run.yaml`](publish-run.yaml) - This example `TaskRun` and
`PipelineResources` demonstrate how to invoke `publish.yaml` (see
[Creating a new release](#creating-a-new-release))

- You can use [`tkn`](https://github.com/tektoncd/cli) to run the [release
pipeline](./release-pipeline.yaml) (see [Creating a new
release](#creating-a-new-release))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops I think https://github.com/tkn/cli is wrong?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lol 😂


#### Setting up your credentials

Expand Down Expand Up @@ -155,49 +182,39 @@ access), such as [the production service account](#production-service-account).

To run the `publish-tekton-pipelines` `Task` and create a release:

1. Pick the revision you want to release and replace the value of the
`PipelineResource` [`tekton-pipelines` `revision`](publish-run.yaml#11),
e.g.:
1. Pick the revision you want to release and update the
[`resources.yaml`](./resources.yaml) file to add a
`PipelineResoruce` for it, e.g.:

```yaml
- name: revision
value: 67efb48746a9d5d7d3b9b5c5cc210de7a47c6ebc # REPLACE with your own commit
apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
name: tekton-pipelines-vX-Y-Z
spec:
type: git
params:
- name: url
value: https://github.com/tektoncd/pipeline # REPLACE with your own fork
- name: revision
value: vX.Y.Z-invalid-tags-boouuhhh # REPLACE with your own commit
```

2. Change the value of the `PipelineRun` [`publish-run`'s `versionTag`
parameter], e.g.:

```yaml
params:
- name: versionTag
value: v0.22222.0 # REPLACE with the version you want to release
```

**TODO(#983) Be careful! if you use a tag that has already been released, you
can overwrite a previous release!**

3. To run against your own infrastructure (not needed for actual releases), also
replace the `imageRegistry` param:

```yaml
- name: imageRegistry
value: gcr.io/tekton-releases # REPLACE with your own registry
```

And the `location` of the `tekton-bucket`:


Also, validate that the `tektoncd-bucket` points to the correct
bucket if you are running the release on your own infrastructure.

```yaml
- name: location
value: gs://tekton-releases # REPLACE with your own bucket
```

4. To run an official release [using the production cluster](https://github.com/tektoncd/plumbing#prow):
2. To run an official release [using the production cluster](https://github.com/tektoncd/plumbing#prow):

```bash
gcloud container clusters get-credentials prow --zone us-central1-a --project tekton-releases
```

5. To run against your own infrastructure (if you are running
3. To run against your own infrastructure (if you are running
[in the production cluster](https://github.com/tektoncd/plumbing#prow) the default account should
already have these creds, this is just a bonus - plus `release-right-meow` might already exist in the
cluster!), also setup the required credentials for the `release-right-meow` service account, either:
Expand Down Expand Up @@ -229,16 +246,68 @@ To run the `publish-tekton-pipelines` `Task` and create a release:
-p "{\"secrets\": [{\"name\": \"$GENERIC_SECRET\"}]}"
```

6. Run the `publish-tekton-pipelines` `Task`:
4. To run the release you can either create a `PipelineRun` using
[`tkn`](https://github.com/tektoncd/cli), or using a yaml file.

```bash
# If you are running in a cluster you've run this in previously, delete the previous run
You will need to set the following parameters:
- `versionTag`: to set the tag to use for published images

**TODO(#983) Be careful! if you use a tag that has already been released, you
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we actually check that? We can fail by default on existing tag, and introduce an optional pipeline parameter force to allow overriding a release in the unlikely case we want to do that - which we never want to do anyways.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely would prefer this to be automatic! #983

can overwrite a previous release!**

- `imageRegistry`: the default value points to
`gcr.io/tekton-releases`, to run against your own infrastructure
(not needed for actual releases) set it to your registry.

6. Run the `release-pipeline`:

```shell
# If you are running in a cluster you've run this in previously,
# delete the previous run and resources

# Apply golang tasks from the catalog
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/master/golang/lint.yaml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: we might have an environment variable to optionally run the release in a dedicated namespace

kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/master/golang/build.yaml
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/master/golang/tests.yaml

# Apply the publish Task
kubectl apply -f tekton/publish.yaml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we run the catalog tasks from github directly, and the tekton ones from local file-system? Those tasks should be executed on the same reference that is in the git resource, or we should mention that one has to locally check-out the version that we are releasing.

kubectl apply -f tekton/publish-run.yaml

# If you are running this in a shared cluster, delete the pipelinerun
# Create the resoruces
kubectl apply -f tekton/resources.yaml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The resources need to be customized (git reference for sure, tekton-bucket might be customized when running a "personal" release.

```

If you are using [`tkn`](https://github.com/tektoncd/cli), you can
run the following command.

```shell
# Do not forget to change those environment variables !
export VERSION_TAG=v0.X.Y

tkn pipeline start \
--param=versionTag=${VERSION_TAG} \
--serviceaccount=release-right-meow \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: Having the service account as an env variable would make it easier to copy/paste this command when running in a local cluster. Unless you want to promote everyone creating a release-right-meow service account, which would be a valid goal to aim for 😸

--resource=source-repo=tekton-pipelines-git \
--resource=bucket=tekton-bucket \
--resource=builtBaseImage=base-image \
--resource=builtEntrypointImage=entrypoint-image \
--resource=builtKubeconfigWriterImage=kubeconfigwriter-image \
--resource=builtCredsInitImage=creds-init-image \
--resource=builtGitInitImage=git-init-image \
--resource=builtNopImage=nop-image \
--resource=builtBashImage=bash-image \
--resource=builtGsutilImage=gsutil-image \
--resource=builtControllerImage=controller-image \
--resource=builtWebhookImage=webhook-image \
--resource=builtDigestExporterImage=digest-exporter-image \
--resource=builtPullRequestInitImage=pull-request-init-image \
pipeline-release
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😻

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't wait until we can create those resources via the command line too! :D

```

If you don't want to use `tkn`, you can use
[`release-pipeline-run.yaml`](./release-pipeline-run.yaml)'s
`PipelineRun`. **Do not forget to update the `params` and the
`source-repo` resource**.

### Authentication

Expand Down
25 changes: 0 additions & 25 deletions tekton/ci-lint-run.yaml

This file was deleted.

71 changes: 0 additions & 71 deletions tekton/ci-lint.yaml

This file was deleted.

28 changes: 0 additions & 28 deletions tekton/ci-unit-test-run.yaml

This file was deleted.

30 changes: 0 additions & 30 deletions tekton/ci-unit-test.yaml

This file was deleted.

Loading