forked from knative/client
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add --service-account flag to cleanup.sh (knative#312)
This is required when running the script from Prow jobs, to establish the authentication. Bonuses: * update documentation * used the term "gcr" instead of project to clearly differentiate from the GCP projects parsed from a YAML file * separated the deletion functions for testing purposes * assorted code refactoring, documentation, simplification, fixing and nitpicking
- Loading branch information
1 parent
5771cc5
commit 528cba9
Showing
5 changed files
with
135 additions
and
135 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,27 +1,34 @@ | ||
| # Resources Clean Up Tool | ||
| This tool is designed to clean up staled resources from gcr, for now it only deletes old images created during testing. | ||
|
|
||
| This tool is designed to clean up stale resources from gcr, for now it only deletes old images created during testing. | ||
|
|
||
| ## Basic Usage | ||
|
|
||
| Directly invoke [cleanup.sh](cleanup.sh) script with certain flags. There is no-op if invoking or sourcing this script without arguments. | ||
|
|
||
| ### Clean up old images from multiple gcr | ||
| By default the current gcloud credentials are used to delete the images. If necessary, use the flag `--service-account _key-file.json_` to specify a service account that will be performing the access to the gcr. | ||
|
|
||
| ### Clean up old images from multiple gcrs | ||
|
|
||
| Projects to be cleaned up are expected to be defined in a `resources.yaml` file. To remove old images from them, call [cleanup.sh](cleanup.sh) with action "delete-old-gcr-images" and following flags: | ||
| - "--project-resource-yaml" as path of `resources.yaml` file - Mandatory | ||
| - "--re-project-name" for regex matching projects names - Optional, default `knative-boskos-[a-zA-Z0-9]+` | ||
| - "--re-project-name" for regex matching projects names - Optional, defaults to `knative-boskos-[a-zA-Z0-9]+` | ||
| - "--days-to-keep" - Optional, default `365` | ||
|
|
||
| Example: | ||
|
|
||
| ```./cleanup.sh "delete-old-gcr-images" --project-resource-yaml "ci/prow/boskos/resources.yaml" --days-to-keep 90``` | ||
|
|
||
| ### Clean up old images from specified gcr | ||
| Cleaning up from specific gcr is supported, except for some special ones (_knative-release_ and _knative-nightly_). Call [cleanup.sh](cleanup.sh) with action "delete-old-gcr-images-from-project" and following flags: | ||
| - "--project-to-cleanup" as name of gcr, e.g. "gcr.io/foo" - Mandatory | ||
| ### Clean up old images from a specific gcr | ||
|
|
||
| Cleaning up from a specific gcr is supported, except for some special ones (_knative-release_ and _knative-nightly_). Call [cleanup.sh](cleanup.sh) with action "delete-old-images-from-gcr" and following flags: | ||
| - "--gcr-to-cleanup" as name of gcr, e.g. "gcr.io/foo" - Mandatory | ||
| - "--days-to-keep" - Optional, default `365` | ||
|
|
||
| Example: | ||
|
|
||
| ```./cleanup.sh "delete-old-gcr-images-from-project" --project-to-cleanup "gcr.io/foo" --days-to-keep 90``` | ||
| ```./cleanup.sh "delete-old-images-from-gcr" --gcr-to-cleanup "gcr.io/foo" --days-to-keep 90``` | ||
|
|
||
| ## Prow Job | ||
|
|
||
| There is a weekly prow job that triggers this tool runs at 11:00/12:00PM(Day light saving) PST every Monday. This tool scans all gcr projects defined in [ci/prow/boskos/resources.yaml](/ci/prow/boskos/resources.yaml) and deletes images older than 90 days. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| #!/bin/bash | ||
|
|
||
| # Copyright 2018 The Knative Authors | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| # Functions for cleaning up GCRs. | ||
| # It doesn't do anything when called from command line. | ||
|
|
||
| source $(dirname $0)/../../scripts/library.sh | ||
|
|
||
| # Delete old images in the given GCR. | ||
| # Parameters: $1 - gcr to be cleaned up (e.g. gcr.io/fooProj) | ||
| # $2 - days to keep images | ||
| function delete_old_images_from_gcr() { | ||
| [[ -z $1 ]] && abort "missing gcr name" | ||
| [[ -z $2 ]] && abort "missing days to keep images" | ||
|
|
||
| is_protected_gcr $1 && \ | ||
| abort "Target GCR set to $1, which is forbidden" | ||
|
|
||
| for image in $(gcloud --format='value(name)' container images list --repository=$1); do | ||
| echo "Checking ${image} for removal" | ||
|
|
||
| delete_old_images_from_gcr ${image} $2 | ||
|
|
||
| local target_date=$(date -d "`date`-$2days" +%Y-%m-%d) | ||
| for digest in $(gcloud --format='get(digest)' container images list-tags ${image} \ | ||
| --filter="timestamp.datetime<${target_date}" --limit=99999); do | ||
| local full_image="${image}@${digest}" | ||
| echo "Deleting image: ${full_image}" | ||
| if (( DRY_RUN )); then | ||
| echo "[DRY RUN] gcloud container images delete -q --force-delete-tags ${full_image}" | ||
| else | ||
| gcloud container images delete -q --force-delete-tags ${full_image} | ||
| fi | ||
| done | ||
| done | ||
| } | ||
|
|
||
| # Delete old images in the GCP projects defined in the yaml file provided. | ||
| # Parameters: $1 - yaml file path defining projects that will be cleaned up | ||
| # $2 - regex pattern for parsing the project names | ||
| # $3 - days to keep images | ||
| function delete_old_gcr_images() { | ||
| [[ -z $1 ]] && abort "missing resource yaml path" | ||
| [[ -z $2 ]] && abort "missing regex pattern for project name" | ||
| [[ -z $3 ]] && abort "missing days to keep images" | ||
|
|
||
| local target_projects # delared here as local + assignment in one line always return 0 exit code | ||
| target_projects="$(grep -Eio "$2" "$1")" | ||
| [[ $? -eq 0 ]] || abort "no project found in $1" | ||
|
|
||
| for project in ${target_projects}; do | ||
| echo "Start deleting images from ${project}" | ||
| delete_old_images_from_gcr "gcr.io/${project}" $3 | ||
| done | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters