From 7441ebe7f6ff59495f295f5555be8a5327bed7c1 Mon Sep 17 00:00:00 2001 From: Christie Wilson Date: Wed, 27 Mar 2019 17:30:40 -0700 Subject: [PATCH] =?UTF-8?q?Update=20release=20dogfood=20to=20work=20agains?= =?UTF-8?q?t=20prod=20=F0=9F=98=8C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The initial iteration of the Task and TaskRun worked just fine against my own infrastructure, but when I tried to get it to work with our production image registry and bucket, I struggled with getting the various tools involved (kaniko, ko and gcloud) to all recognize the service account. In the end the easiest way to make this work was to lean on gcloud and manually mounting the secrets into each Task. I still attach the secret to the service account that runs the TaskRun because I think that's needed for the bucket output uploading at the end. The other goal here is to document this process so that someone else can run the release Task next time :D (Though eventually we want to be able to trigger this automatically to run in a different cluster, probably `tekton-releases` own cluster.) Two remaining unknowns that I'm hoping we can explore later: - Could I have avoided using gcloud to auth by using servie accounts and secrets with annotations properly (https://github.com/tektoncd/pipeline/blob/master/docs/auth.md#guiding-credential-selection) - Could I have avoided manually mounting the secrets by annotating the secret in some manner that would result in it being automatically mounted? --- docs/install.md | 8 +++- tekton/README.md | 99 ++++++++++++++++++++++++++++++++++++++--- tekton/account.yaml | 4 ++ tekton/publish-run.yaml | 1 + tekton/publish.yaml | 34 ++++++++++++-- 5 files changed, 134 insertions(+), 12 deletions(-) create mode 100644 tekton/account.yaml diff --git a/docs/install.md b/docs/install.md index b831bca0f09..de5fbba9a7d 100644 --- a/docs/install.md +++ b/docs/install.md @@ -13,11 +13,13 @@ Use this page to add the component to an existing Kubernetes cluster. ``` 2. Grant cluster-admin permissions to the current user: + ```bash kubectl create clusterrolebinding cluster-admin-binding \ --clusterrole=cluster-admin \ --user=$(gcloud config get-value core/account) ``` + _See [Role-based access control](https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control#prerequisites_for_using_role-based_access_control) for more information_. @@ -32,9 +34,11 @@ To add the Tekton Pipelines component to an existing cluster: and its dependencies: ```bash - kubectl apply --filename https://storage.googleapis.com/knative-releases/build-pipeline/latest/release.yaml + kubectl apply --filename https://storage.googleapis.com/tekton-releases/latest/release.yaml ``` + _(Previous versions will be available at `previous/$VERSION_NUMBER`, e.g. https://storage.googleapis.com/tekton-releases/previous/0.2.0/release.yaml.)_ + 1. Run the [`kubectl get`](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get) command to monitor the Tekton Pipelines components until all of the @@ -82,7 +86,7 @@ the persistent volume can fail. ## Custom Releases -The [release script](./../hack/release.md) can be used for creating a custom +The [relase Task](./../tekton/README.md) can be used for creating a custom release of Tekton Pipelines. This can be useful for advanced users that need to configure the container images built and used by the Pipelines components. diff --git a/tekton/README.md b/tekton/README.md index 2b089a91ac9..2f694e07b07 100644 --- a/tekton/README.md +++ b/tekton/README.md @@ -30,8 +30,8 @@ The `Tasks` which make up our release `Pipeline` are: ### Running -To run these `Pipelines` and `Tasks`, you must have Tekton Pipelines installed, -either via +To run these `Pipelines` and `Tasks`, you must have Tekton Pipelines installed +(in your own kubernetes cluster) either via [an official release](https://github.com/knative/build-pipeline/blob/master/docs/install.md) or [from `HEAD`](https://github.com/knative/build-pipeline/blob/master/DEVELOPMENT.md#install-pipeline). @@ -56,12 +56,87 @@ image registry instead. ``` - [`publish-run.yaml`](publish-run.yaml) - This example `TaskRun` and - `PipelineResources` demonstrate how to invoke `publish.yaml`: + `PipelineResources` demonstrate how to invoke `publish.yaml` (see [Creating a new release](#creating-a-new-release)) - ```bash - kubectl apply -f tekton/publish.yaml - kubectl apply -f tekton/publish-run.yaml - ``` +#### Creating a new release + +The `TaskRun` will use + +- The kubernetes service account [`release-right-meow`](account.yaml), which by + default has no associated secrets +- A secret called `release-secret` + +It needs to run with a service account in the target GCP project with +[`Storage Admin`](https://cloud.google.com/container-registry/docs/access-control) +access). + +To run the `publish-tekton-pipelines` `Task` and create a release: + +1. Pick the revision you want to release and replace the value of the `PipelineResouce` + [`tekton-pipelines` `revision`](publish-run.yaml#11), e.g.: + + ```yaml + - name: revision + value: 67efb48746a9d5d7d3b9b5c5cc210de7a47c6ebc # REPLACE with your own commit + ``` + +2. Change the value of the `PipelineRun` [`publish-run`'s `versionTag` parameter], e.g.: + + ```yaml + params: + - name: versionTag + value: 0.2.0 # REPLACE with the version you want to release + ``` + +3. To run against your own infrastructure (not needed for actual releases), also replace + the `imageRegistry` param: + + ```yaml + - name: imageRegistry + value: gcr.io/tekton-releases # REPLACE with your own registry + ``` + + And the `location` of the `tekton-bucket`: + + ```yaml + - name: location + value: gs://tekton-releases # REPLACE with your own bucket + ``` + +4. Setup the required credentials for the `release-right-meow` service acount, either: + + - For [the GCP service account + `release-right-meow@tekton-releases.iam.gserviceaccount.com`](#production-service-account) + which has the proper authorization to release the images and yamls in + [our `tekton-releases` GCP project](../infra/README.md#prow) + - For [your own GCP service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) + if running against your own infrastructure + + ```bash + KEY_FILE=release.json + GENERIC_SECRET=release-secret + ACCOUNT=release-right-meow + # Replace with your own service account if using your own infra + GCP_ACCOUNT="release-right-meow@tekton-releases.iam.gserviceaccount.com" + + # 1. Create a private key for the service account, which you can use + gcloud iam service-accounts keys create --iam-account $GCP_ACCOUNT $KEY_FILE + + # 2. Create kubernetes secret, which we will use via a service account and directly mounting + kubectl create secret generic $GENERIC_SECRET --from-file=./$KEY_FILE + + # 3. Add the docker secret to the service account + kubectl apply -f tekton/account.yaml + kubectl patch serviceaccount $ACCOUNT \ + -p "{\"secrets\": [{\"name\": \"$GENERIC_SECRET\"}]}" + ``` + +5. Run the `publish-tekton-pipelines` `Task`: + + ```bash + kubectl apply -f tekton/publish.yaml + kubectl apply -f tekton/publish-run.yaml + ``` ### Authentication @@ -70,6 +145,7 @@ Users executing the publish task must be able to: - Push to the image registry (production registry is `gcr.io/tekton-releases`) - Write to the GCS bucket (production bucket is `gs://tekton-releases`) +TODO: To be able to publish images via `kaniko` or `ko`, you must be able to push to your image registry. At the moment, the publish `Task` will try to use your default service account in the namespace where you create the `TaskRun`. If that @@ -90,6 +166,15 @@ For now, users who need access to our production registry (`gcr.io/tekton-releases`) and production GCS bucket (`gs://tekton-releases`) should ping @bobcatfish or @dlorenc to get added to the authorized users. +##### Production service account + +TODO(christiewilson, dlorenc): add a group which has access to this service account + +The GCP service account for creating release is +`release-right-meow@tekton-releases.iam.gserviceaccount.com`. This account has the role +[`Storage Admin`](https://cloud.google.com/container-registry/docs/access-control) +in order to be able to read and write buckets and images. + ## Supporting scripts Some supporting scripts have been written using Python 2.7: diff --git a/tekton/account.yaml b/tekton/account.yaml new file mode 100644 index 00000000000..720059ead8e --- /dev/null +++ b/tekton/account.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: release-right-meow \ No newline at end of file diff --git a/tekton/publish-run.yaml b/tekton/publish-run.yaml index ad14933c548..cc846e0e0c3 100644 --- a/tekton/publish-run.yaml +++ b/tekton/publish-run.yaml @@ -129,6 +129,7 @@ kind: TaskRun metadata: name: publish-run spec: + serviceAccount: release-right-meow taskRef: name: publish-tekton-pipelines trigger: diff --git a/tekton/publish.yaml b/tekton/publish.yaml index af5f52014ff..6939949d8c2 100644 --- a/tekton/publish.yaml +++ b/tekton/publish.yaml @@ -52,6 +52,13 @@ spec: - --destination=${inputs.params.imageRegistry}/${inputs.params.pathToProject}/${outputs.resources.builtBaseImage.url} - --context=/workspace/go/src/github.com/tektoncd/pipeline + volumeMounts: + - name: gcp-secret + mountPath: /secret + env: + - name: GOOGLE_APPLICATION_CREDENTIALS + value: /secret/release.json + - name: create-ko-yaml image: busybox command: @@ -63,7 +70,7 @@ spec: set -x cat < /workspace/go/src/github.com/tektoncd/pipeline/.ko.yaml - # By default `ko` will build images on top of `distroless` + # By default ko will build images on top of distroless baseImageOverrides: ${inputs.params.pathToProject}/${outputs.resources.builtCredsInitImage.url}: ${inputs.params.imageRegistry}/${inputs.params.pathToProject}/build-base:latest ${inputs.params.pathToProject}/${outputs.resources.builtGitInitImage.url}: ${inputs.params.imageRegistry}/${inputs.params.pathToProject}/build-base:latest @@ -90,6 +97,8 @@ spec: value: ${inputs.params.imageRegistry} - name: GOPATH value: /workspace/go + - name: CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE + value: /secret/release.json command: - /bin/sh args: @@ -98,14 +107,20 @@ spec: set -e set -x - # TODO(#631) This is a hack to auth with the default creds, need a solution that supports service accounts + # Auth with CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE gcloud auth configure-docker # ko requires this variable to be set in order to set image creation timestamps correctly https://github.com/google/go-containerregistry/pull/146 export SOURCE_DATE_EPOCH=`date +%s` + # Change to directory with our .ko.yaml + cd /workspace/go/src/github.com/tektoncd/pipeline + # Publish images and create release.yaml ko resolve --preserve-import-paths -f /workspace/go/src/github.com/tektoncd/pipeline/config/ > /workspace/bucket/latest/release.yaml + volumeMounts: + - name: gcp-secret + mountPath: /secret - name: copy-to-tagged-bucket image: busybox @@ -140,8 +155,13 @@ spec: ${inputs.params.imageRegistry}/${inputs.params.pathToProject}/${outputs.resources.builtControllerImage.url} ${inputs.params.imageRegistry}/${inputs.params.pathToProject}/${outputs.resources.builtWebhookImage.url} ) + # Parse the built images from the release.yaml generated by ko BUILT_IMAGES=( $(/workspace/go/src/github.com/tektoncd/pipeline/tekton/koparse/koparse.py --path /workspace/bucket/latest/release.yaml --base ${inputs.params.imageRegistry}/${inputs.params.pathToProject} --images ${IMAGES[@]}) ) + # Auth with account credentials + gcloud auth activate-service-account --key-file=/secret/release.json + + # Tag the images and put them in all the regions for IMAGE in "${BUILT_IMAGES[@]}" do for REGION in "${REGIONS[@]}" @@ -152,4 +172,12 @@ spec: gcloud -q container images add-tag ${IMAGE} ${REGION}.${IMAGE_WITHOUT_SHA}:$TAG done done - done \ No newline at end of file + done + volumeMounts: + - name: gcp-secret + mountPath: /secret + + volumes: + - name: gcp-secret + secret: + secretName: release-secret \ No newline at end of file