][share_email]
[
][share_googleplus]
[
][share_facebook]
@@ -73,7 +55,6 @@ It's 100% Open Source and licensed under the [APACHE2](LICENSE).
-
## Screenshots
@@ -82,6 +63,9 @@ It's 100% Open Source and licensed under the [APACHE2](LICENSE).
+
+
+
## Usage
@@ -104,9 +88,9 @@ Run `make help` for a list of available targets.
The `build-harness` is compatible with [GitHub Actions](https://github.com/features/actions).
-Here's an example of running `make readme/lint`
+Here's an example of running `make readme/lint`
-```
+```yaml
name: build-harness/readme/lint
on: [pull_request]
jobs:
@@ -139,8 +123,9 @@ Here are some real world examples:
+
## Makefile Targets
-```
+```text
Available targets:
aws/install Install aws cli bundle
@@ -148,6 +133,7 @@ Available targets:
bash/lint Lint all bash scripts
chamber/install Install chamber
chamber/shell Start a chamber shell with secrets exported to the environment
+ clean Clean build-harness
codefresh/export DEPRECATED!!! Export codefresh additional envvars
codefresh/notify/slack/build Send notification from codefresh to slack using "build" template
codefresh/notify/slack/deploy Send notification from codefresh to slack using "deploy" template
@@ -236,6 +222,7 @@ Available targets:
help Help screen
help/all Display help for all targets
help/short This help short screen
+ init Init build-harness
jenkins/run-job-with-tag Run a Jenkins Job with $(TAG)
make/lint Lint all makefiles
packages/delete Delete packages
@@ -254,32 +241,81 @@ Available targets:
slack/notify/deploy Send notification to slack using "deploy" template
template/build Create $OUT file by building it from $IN template file
template/deps Install dependencies
+ terraform/bump-tf-12-min-version Rewrite versions.tf to bump modules with minimum core version of '0.12.x' to '>= 0.12.26'
terraform/get-modules Ensure all modules can be fetched
terraform/get-plugins Ensure all plugins can be fetched
terraform/install Install terraform
terraform/lint Lint check Terraform
- terraform/upgrade-modules Upgrade all terraform module sources
+ terraform/loosen-constraints and convert "~>" constraints to ">=".
+ terraform/rewrite-required-providers Rewrite versions.tf to update existing configuration to add an explicit source attribute for each provider
+ terraform/upgrade-modules This target has not been upgraded to handle registry format
terraform/validate Basic terraform sanity check
travis/docker-login Login into docker hub
travis/docker-tag-and-push Tag & Push according Travis environment variables
```
+
+
## Extending `build-harness` with targets from another repo
It is possible to extend the `build-harness` with targets and entire modules of your own, without having to fork or modify `build-harness` itself.
This might be useful if, for example, you wanted to maintain some tooling that was specific to your environment that didn't have enough general applicability to be part of the main project.
This makes it so you don't necessarily need to fork `build-harness` itself - you can place a repo defined by the environment variable `BUILD_HARNESS_EXTENSIONS_PATH` (a filesystem peer of `build-harness` named `build-harness-extensions` by default) and populate it with tools in the same `Makefile` within `module` structure as `build-harness` has.
Modules will be combined and available with a unified `make` command.
+
+
+## Using the "auto-init" feature
+
+Typically, the `build-harness` project requires running `make init` before any of the Makefile targets can be invoked. The `init` target will "install" the `build-harness` project and "include" the `Makefile` from the `build-harness` project.
+
+Alternatively, the "auto-init" feature can automatically run the `init` logic for you to install the `build-harness` and help keep the install up-to-date. This feature is enabled using the env or Makefile variable `BUILD_HARNESS_AUTO_INIT=true`. By default, this feature is disabled; to enable it, you must set the variable yourself.
+**Note:** The "auto-init" feature is a convenience for running `make` interactively. Regardless of your setting of `BUILD_HARNESS_AUTO_INIT`, "auto-init" will be disabled if `make` is running inside a Docker container. Scripts and automation should continue to call `make init` explicitly.
+```make
+BUILD_HARNESS_AUTO_INIT = true
-## Share the Love
+-include $(shell curl -sSL -o .build-harness "https://git.io/build-harness"; echo .build-harness)
+```
-Like this project? Please give it a ★ on [our GitHub](https://github.com/cloudposse/build-harness)! (it helps us **a lot**)
+The "auto-init" feature will _also_ keep the install up-to-date. It will check the value of `BUILD_HARNESS_BRANCH`, get the commit ID, compare that to the current checkout, and update the clone if they differ. A useful side-effect is that it becomes easy to pin to versions of the `build-harness` from your own project, and let the `build-harness` update itself as you update the pin:
+
+```make
+BUILD_HARNESS_AUTO_INIT = true
+BUILD_HARNESS_BRANCH = {TAG}
+
+-include $(shell curl -sSL -o .build-harness "https://git.io/build-harness"; echo .build-harness)
+```
+
+Now when you run `make` the project will update itself to use the version specified by the `BUILD_HARNESS_BRANCH` value:
+
+```sh
+$ make help
+Removing existing build-harness
+Cloning https://github.com/cloudposse/build-harness.git#{TAG}...
+Cloning into 'build-harness'...
+remote: Enumerating objects: 143, done.
+remote: Counting objects: 100% (143/143), done.
+remote: Compressing objects: 100% (118/118), done.
+remote: Total 143 (delta 7), reused 71 (delta 3), pack-reused 0
+Receiving objects: 100% (143/143), 85.57 KiB | 2.09 MiB/s, done.
+Resolving deltas: 100% (7/7), done.
+Available targets:
+
+ aws/install Install aws cli bundle
+```
+
+
+
+
+## Share the Love
+
+Like this project? Please give it a ★ on [our GitHub](https://github.com/cloudposse/build-harness)! (it helps us **a lot**)
Are you using this project or any of our other projects? Consider [leaving a testimonial][testimonial]. =)
+
## Related Projects
Check out these related projects.
@@ -288,18 +324,16 @@ Check out these related projects.
- [Dev Harness](https://github.com/cloudposse/dev) - Cloud Posse Local Development Harness
-
-
## References
-For additional context, refer to some of these links.
+For additional context, refer to some of these links.
- [Wikipedia - Test Harness](https://en.wikipedia.org/wiki/Test_harness) - The `build-harness` is similar in concept to a "Test Harness"
## Help
-**Got a question?** We got answers.
+**Got a question?** We got answers.
File a GitHub [issue](https://github.com/cloudposse/build-harness/issues), send us an [email][email] or join our [Slack Community][slack].
@@ -308,7 +342,7 @@ File a GitHub [issue](https://github.com/cloudposse/build-harness/issues), send
## DevOps Accelerator for Startups
-We are a [**DevOps Accelerator**][commercial_support]. We'll help you build your cloud infrastructure from the ground up so you can own it. Then we'll show you how to operate it and stick around for as long as you need us.
+We are a [**DevOps Accelerator**][commercial_support]. We'll help you build your cloud infrastructure from the ground up so you can own it. Then we'll show you how to operate it and stick around for as long as you need us.
[][commercial_support]
@@ -331,13 +365,17 @@ We deliver 10x the value for a fraction of the cost of a full-time engineer. Our
Join our [Open Source Community][slack] on Slack. It's **FREE** for everyone! Our "SweetOps" community is where you get to talk with others who share a similar vision for how to rollout and manage infrastructure. This is the best place to talk shop, ask questions, solicit feedback, and work together as a community to build totally *sweet* infrastructure.
+## Discourse Forums
+
+Participate in our [Discourse Forums][discourse]. Here you'll find answers to commonly asked questions. Most questions will be related to the enormous number of projects we support on our GitHub. Come here to collaborate on answers, find solutions, and get ideas about the products and services we value. It only takes a minute to get started! Just sign in with SSO using your GitHub account.
+
## Newsletter
-Sign up for [our newsletter][newsletter] that covers everything on our technology radar. Receive updates on what we're up to on GitHub as well as awesome new projects we discover.
+Sign up for [our newsletter][newsletter] that covers everything on our technology radar. Receive updates on what we're up to on GitHub as well as awesome new projects we discover.
## Office Hours
-[Join us every Wednesday via Zoom][office_hours] for our weekly "Lunch & Learn" sessions. It's **FREE** for everyone!
+[Join us every Wednesday via Zoom][office_hours] for our weekly "Lunch & Learn" sessions. It's **FREE** for everyone!
[![zoom](https://img.cloudposse.com/fit-in/200x200/https://cloudposse.com/wp-content/uploads/2019/08/Powered-by-Zoom.png")][office_hours]
@@ -365,34 +403,36 @@ In general, PRs are welcome. We follow the typical "fork-and-pull" Git workflow.
## Copyrights
-Copyright © 2016-2020 [Cloud Posse, LLC](https://cloudposse.com)
+Copyright © 2016-2021 [Cloud Posse, LLC](https://cloudposse.com)
-## License
+## License
-[](https://opensource.org/licenses/Apache-2.0)
+[](https://opensource.org/licenses/Apache-2.0)
See [LICENSE](LICENSE) for full details.
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- https://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
+```text
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+```
@@ -414,7 +454,7 @@ This project is maintained and funded by [Cloud Posse, LLC][website]. Like it? P
We're a [DevOps Professional Services][hire] company based in Los Angeles, CA. We ❤️ [Open Source Software][we_love_open_source].
-We offer [paid support][commercial_support] on all of our projects.
+We offer [paid support][commercial_support] on all of our projects.
Check out [our other projects][github], [follow us on twitter][twitter], [apply for a job][jobs], or [hire us][hire] to help with your cloud strategy and implementation.
@@ -422,8 +462,10 @@ Check out [our other projects][github], [follow us on twitter][twitter], [apply
### Contributors
-| [![Erik Osterman][osterman_avatar]][osterman_homepage][Erik Osterman][osterman_homepage] | [![Igor Rodionov][goruha_avatar]][goruha_homepage]
[Igor Rodionov][goruha_homepage] | [![Andriy Knysh][aknysh_avatar]][aknysh_homepage]
[Andriy Knysh][aknysh_homepage] | [![Sarkis][sarkis_avatar]][sarkis_homepage]
[Sarkis][sarkis_homepage] | [![Alexander Babai][alebabai_avatar]][alebabai_homepage]
[Alexander Babai][alebabai_homepage] | [![Jon Boulle][jonboulle_avatar]][jonboulle_homepage]
[Jon Boulle][jonboulle_homepage] | -|---|---|---|---|---|---| + +| [![Erik Osterman][osterman_avatar]][osterman_homepage]
[Erik Osterman][osterman_homepage] | [![Igor Rodionov][goruha_avatar]][goruha_homepage]
[Igor Rodionov][goruha_homepage] | [![Andriy Knysh][aknysh_avatar]][aknysh_homepage]
[Andriy Knysh][aknysh_homepage] | [![Sarkis][sarkis_avatar]][sarkis_homepage]
[Sarkis][sarkis_homepage] | [![Alexander Babai][alebabai_avatar]][alebabai_homepage]
[Alexander Babai][alebabai_homepage] | [![Jon Boulle][jonboulle_avatar]][jonboulle_homepage]
[Jon Boulle][jonboulle_homepage] | [![Marcin Brański][3h4x_avatar]][3h4x_homepage]
[Marcin Brański][3h4x_homepage] | +|---|---|---|---|---|---|---| + [osterman_homepage]: https://github.com/osterman [osterman_avatar]: https://img.cloudposse.com/150x150/https://github.com/osterman.png @@ -437,6 +479,8 @@ Check out [our other projects][github], [follow us on twitter][twitter], [apply [alebabai_avatar]: https://img.cloudposse.com/150x150/https://github.com/alebabai.png [jonboulle_homepage]: https://github.com/jonboulle [jonboulle_avatar]: https://img.cloudposse.com/150x150/https://github.com/jonboulle.png + [3h4x_homepage]: https://github.com/3h4x + [3h4x_avatar]: https://img.cloudposse.com/150x150/https://github.com/3h4x.png [![README Footer][readme_footer_img]][readme_footer_link] [![Beacon][beacon]][website] @@ -453,6 +497,7 @@ Check out [our other projects][github], [follow us on twitter][twitter], [apply [testimonial]: https://cpco.io/leave-testimonial?utm_source=github&utm_medium=readme&utm_campaign=cloudposse/build-harness&utm_content=testimonial [office_hours]: https://cloudposse.com/office-hours?utm_source=github&utm_medium=readme&utm_campaign=cloudposse/build-harness&utm_content=office_hours [newsletter]: https://cpco.io/newsletter?utm_source=github&utm_medium=readme&utm_campaign=cloudposse/build-harness&utm_content=newsletter + [discourse]: https://ask.sweetops.com/?utm_source=github&utm_medium=readme&utm_campaign=cloudposse/build-harness&utm_content=discourse [email]: https://cpco.io/email?utm_source=github&utm_medium=readme&utm_campaign=cloudposse/build-harness&utm_content=email [commercial_support]: https://cpco.io/commercial-support?utm_source=github&utm_medium=readme&utm_campaign=cloudposse/build-harness&utm_content=commercial_support [we_love_open_source]: https://cpco.io/we-love-open-source?utm_source=github&utm_medium=readme&utm_campaign=cloudposse/build-harness&utm_content=we_love_open_source diff --git a/README.yaml b/README.yaml index c155efcc2..f31bfceb9 100644 --- a/README.yaml +++ b/README.yaml @@ -1,11 +1,11 @@ --- -# +# # This is the canonical configuration for the `README.md` # Run `make readme` to rebuild the `README.md` # # Name of this project -name: Build Harness +name: Build Harness # Logo for this project #logo: docs/logo.png @@ -19,20 +19,23 @@ copyrights: url: "https://cloudposse.com" year: "2016" -# Canonical GitHub repo +# Canonical GitHub repo github_repo: cloudposse/build-harness # Badges to display badges: - name: "Build Status" - image: "https://travis-ci.org/cloudposse/build-harness.svg?branch=master" - url: "https://travis-ci.org/cloudposse/build-harness" + image: "https://github.com/cloudposse/build-harness/workflows/docker/badge.svg?branch=master" + url: "https://github.com/cloudposse/build-harness/actions?query=workflow%3Adocker" - name: "Latest Release" image: "https://img.shields.io/github/release/cloudposse/build-harness.svg" url: "https://github.com/cloudposse/build-harness/releases/latest" - name: "Slack Community" image: "https://slack.cloudposse.com/badge.svg" url: "https://slack.cloudposse.com" + - name: "Discourse Forum" + image: "https://img.shields.io/discourse/https/ask.sweetops.com/posts.svg" + url: "https://ask.sweetops.com/" related: - name: "Packages" @@ -53,7 +56,7 @@ screenshots: - name: "demo" description: "Example of using the `build-harness` to build a docker image" url: "https://cdn.rawgit.com/cloudposse/build-harness/master/docs/demo.svg" - + # Short description of this project description: |- This `build-harness` is a collection of Makefiles to facilitate building Golang projects, Dockerfiles, Helm charts, and more. @@ -80,12 +83,12 @@ usage: |- **NOTE:** the `/` is interchangable with the `:` in target names ## GitHub Actions - + The `build-harness` is compatible with [GitHub Actions](https://github.com/features/actions). - Here's an example of running `make readme/lint` + Here's an example of running `make readme/lint` - ``` + ```yaml name: build-harness/readme/lint on: [pull_request] jobs: @@ -119,6 +122,7 @@ quickstart: |- include: - "docs/targets.md" - "docs/extensions.md" + - "docs/auto-init.md" # Contributors to this project contributors: @@ -134,3 +138,5 @@ contributors: github: "alebabai" - name: "Jon Boulle" github: "jonboulle" + - name: "Marcin Brański" + github: "3h4x" diff --git a/bin/generate_related_references.py b/bin/generate_related_references.py new file mode 100755 index 000000000..d59c5de14 --- /dev/null +++ b/bin/generate_related_references.py @@ -0,0 +1,113 @@ +#!/usr/bin/env python3 + +import json +import os +import subprocess + +import requests +from github import Github +from iteration_utilities import unique_everseen +from ruamel.yaml import YAML + +GH_TOKEN = os.environ["GITHUB_TOKEN"] +GH_ORG_NAME = os.getenv("GH_ORG_NAME", "cloudposse") +GH_SEARCH_PATTERN = os.getenv("GH_SEARCH_PATTERN", "terraform-") +TF_MODULE_PATH = os.getenv("TF_MODULE_PATH", ".") +TF_CONFIG_INSPECT_BINARY_PATH = os.getenv( + "TF_CONFIG_INSPECT_BINARY_PATH", "terraform-config-inspect" +) +TF_REGISTRY_URL = "https://registry.terraform.io" + +gh = Github(GH_TOKEN) +yaml = YAML(typ="rt") +yaml.default_flow_style = False +yaml.preserve_quotes = False + + +def parse_gh(): + gh_repos = [] + for repo in gh.get_organization(GH_ORG_NAME).get_repos(): + if GH_SEARCH_PATTERN in repo.name: + repo_object = {} + repo_object["name"] = repo.name + repo_object["description"] = repo.description + repo_object["url"] = repo.html_url + gh_repos.append(repo_object) + return gh_repos + + +def tf_config_inspect(): + output = json.loads( + subprocess.check_output( + [TF_CONFIG_INSPECT_BINARY_PATH, TF_MODULE_PATH, "--json"], + stderr=subprocess.STDOUT, + ) + ) + return output + + +def parse_tf_registry(src_data, src_type): + items = [] + src_item = "module_calls" + if src_type == "providers": + src_item = "required_providers" + + for k, v in src_data[src_item].items(): + item_object = {} + url = TF_REGISTRY_URL + "/v1/" + src_type + "/" + v["source"] + r = requests.get(url=url).json() + + if src_type == "providers": + name_pattern = "terraform-provider-{}".format(r["name"]) + else: + name_pattern = "terraform-{}-{}".format(r["provider"], r["name"]) + item_object["name"] = name_pattern + + if src_type == "providers": + # description on GitHub looks better than on terraform-registry + gh_repo_info = gh.get_repo("{}/{}".format(r["namespace"], name_pattern)) + item_object["description"] = gh_repo_info.description + item_object["url"] = TF_REGISTRY_URL + "/providers/{}/{}/latest".format( + r["namespace"], r["name"] + ) + else: + item_object["description"] = r["description"] + item_object["url"] = r["source"] + + items.append(item_object) + return items + + +if __name__ == "__main__": + related_list = [] + reference_list = [] + + inspected_data = tf_config_inspect() + modules_list = parse_tf_registry(inspected_data, "modules") + providers_list = parse_tf_registry(inspected_data, "providers") + gh_repos_list = parse_gh() + + # this can be done in one line but it requires itertools + # and additional step to remove empty dicts + for m in unique_everseen(modules_list): + related_list.append(m) + for g in unique_everseen(gh_repos_list): + related_list.append(g) + for p in unique_everseen(providers_list): + reference_list.append(p) + + with open("{}/README.yaml".format(TF_MODULE_PATH)) as f: + readme = yaml.load(f) + + readme["related"] = related_list + + # ensure that "references" key is present and then insert data + if readme.get("references"): + readme["references"] = reference_list + else: + # create key "references" after the "related" + readme.insert(list(readme.keys()).index("related") + 1, "references", []) + readme["references"] = reference_list + + with open("{}/README.yaml".format(TF_MODULE_PATH), "w") as f: + yaml.dump(readme, f) diff --git a/bin/install.sh b/bin/install.sh index 1fef0edf3..9caaa9877 100755 --- a/bin/install.sh +++ b/bin/install.sh @@ -10,4 +10,4 @@ if [ "$BUILD_HARNESS_PROJECT" ] && [ -d "$BUILD_HARNESS_PROJECT" ]; then fi echo "Cloning ${GITHUB_REPO}#${BUILD_HARNESS_BRANCH}..." -git clone -b $BUILD_HARNESS_BRANCH $GITHUB_REPO +git clone -c advice.detachedHead=false --depth=1 -b $BUILD_HARNESS_BRANCH $GITHUB_REPO diff --git a/bin/terraform-docs.awk b/bin/terraform-docs.awk deleted file mode 100644 index bd6b2b7ff..000000000 --- a/bin/terraform-docs.awk +++ /dev/null @@ -1,90 +0,0 @@ -# This script converts Terraform 0.12 variables/outputs to something suitable for `terraform-docs` -# As of terraform-docs v0.6.0, HCL2 is not supported. This script is a *dirty hack* to get around it. -# https://github.com/segmentio/terraform-docs/ -# https://github.com/segmentio/terraform-docs/issues/62 - -{ - if ( $0 ~ /\{/ ) { - braceCnt++ - } - - if ( $0 ~ /\}/ ) { - braceCnt-- - } - - # [START] variable or output block started - if ($0 ~ /^[[:space:]]*(variable|output)[[:space:]][[:space:]]*"(.*?)"/) { - # Normalize the braceCnt (should be 1 now) - braceCnt = 1 - # [CLOSE] "default" block - if (blockDefCnt > 0) { - blockDefCnt = 0 - } - blockCnt++ - print $0 - } - - # [START] multiline default statement started - if (blockCnt > 0) { - if ($0 ~ /^[[:space:]][[:space:]]*(default)[[:space:]][[:space:]]*=/) { - if ($3 ~ "null") { - print " default = \"null\"" - } else { - print $0 - blockDefCnt++ - blockDefStart=1 - } - } - } - - # [PRINT] single line "description" - if (blockCnt > 0) { - if (blockDefCnt == 0) { - if ($0 ~ /^[[:space:]][[:space:]]*description[[:space:]][[:space:]]*=/) { - # [CLOSE] "default" block - if (blockDefCnt > 0) { - blockDefCnt = 0 - } - print $0 - } - } - } - - # [PRINT] single line "type" - if (blockCnt > 0) { - if ($0 ~ /^[[:space:]][[:space:]]*type[[:space:]][[:space:]]*=/ ) { - # [CLOSE] "default" block - if (blockDefCnt > 0) { - blockDefCnt = 0 - } - type=$3 - if (type ~ "object") { - print " type = \"object\"" - } else { - # legacy quoted types: "string", "list", and "map" - if ($3 ~ /^[[:space:]]*"(.*?)"[[:space:]]*$/) { - print " type = " $3 - } else { - print " type = \"" $3 "\"" - } - } - } - } - - # [CLOSE] variable/output block - if (blockCnt > 0) { - if (braceCnt == 0 && blockCnt > 0) { - blockCnt-- - print $0 - } - } - - # [PRINT] Multiline "default" statement - if (blockCnt > 0 && blockDefCnt > 0) { - if (blockDefStart == 1) { - blockDefStart = 0 - } else { - print $0 - } - } -} diff --git a/bin/terraform-docs.sh b/bin/terraform-docs.sh deleted file mode 100755 index 4e6be996d..000000000 --- a/bin/terraform-docs.sh +++ /dev/null @@ -1,14 +0,0 @@ -#!/bin/bash - -which awk 2>&1 >/dev/null || ( echo "awk not available"; exit 1) -which terraform 2>&1 >/dev/null || ( echo "terraform not available"; exit 1) -which terraform-docs 2>&1 >/dev/null || ( echo "terraform-docs not available"; exit 1) - -if [[ "`terraform version | head -1`" =~ 0\.12 ]]; then - TMP_FILE="$(mktemp /tmp/terraform-docs-XXXXXXXXXX)" - awk -f ${BUILD_HARNESS_PATH}/bin/terraform-docs.awk $2/*.tf > ${TMP_FILE} - terraform-docs $1 ${TMP_FILE} - rm -f ${TMP_FILE} -else - terraform-docs $1 $2 -fi diff --git a/bin/upgrade_terraform_modules.sh b/bin/upgrade_terraform_modules.sh index cd596682a..f7c27133a 100755 --- a/bin/upgrade_terraform_modules.sh +++ b/bin/upgrade_terraform_modules.sh @@ -18,7 +18,7 @@ function github_latest_release() { function upgrade_modules() { local file=$1 echo "Processing $file..." - for source in $(json2hcl -reverse < $file | jq -r '.module | .[][] | first | .source' 2>/dev/null); do + for source in $(grep -Po '^\s*source\s*=\s*"(.*?)"' -r .|cut -d'"' -f2|sort -u); do if [[ $source =~ github.com/ ]]; then echo "[GITHUB]: $source" if [[ $source =~ github.com/(.*?)/(.*?)\.git ]]; then diff --git a/docs/auto-init.md b/docs/auto-init.md new file mode 100644 index 000000000..81ed1e117 --- /dev/null +++ b/docs/auto-init.md @@ -0,0 +1,42 @@ + +## Using the "auto-init" feature + +Typically, the `build-harness` project requires running `make init` before any of the Makefile targets can be invoked. The `init` target will "install" the `build-harness` project and "include" the `Makefile` from the `build-harness` project. + +Alternatively, the "auto-init" feature can automatically run the `init` logic for you to install the `build-harness` and help keep the install up-to-date. This feature is enabled using the env or Makefile variable `BUILD_HARNESS_AUTO_INIT=true`. By default, this feature is disabled; to enable it, you must set the variable yourself. + +**Note:** The "auto-init" feature is a convenience for running `make` interactively. Regardless of your setting of `BUILD_HARNESS_AUTO_INIT`, "auto-init" will be disabled if `make` is running inside a Docker container. Scripts and automation should continue to call `make init` explicitly. + +```make +BUILD_HARNESS_AUTO_INIT = true + +-include $(shell curl -sSL -o .build-harness "https://git.io/build-harness"; echo .build-harness) +``` + +The "auto-init" feature will _also_ keep the install up-to-date. It will check the value of `BUILD_HARNESS_BRANCH`, get the commit ID, compare that to the current checkout, and update the clone if they differ. A useful side-effect is that it becomes easy to pin to versions of the `build-harness` from your own project, and let the `build-harness` update itself as you update the pin: + +```make +BUILD_HARNESS_AUTO_INIT = true +BUILD_HARNESS_BRANCH = {TAG} + +-include $(shell curl -sSL -o .build-harness "https://git.io/build-harness"; echo .build-harness) +``` + +Now when you run `make` the project will update itself to use the version specified by the `BUILD_HARNESS_BRANCH` value: + +```sh +$ make help +Removing existing build-harness +Cloning https://github.com/cloudposse/build-harness.git#{TAG}... +Cloning into 'build-harness'... +remote: Enumerating objects: 143, done. +remote: Counting objects: 100% (143/143), done. +remote: Compressing objects: 100% (118/118), done. +remote: Total 143 (delta 7), reused 71 (delta 3), pack-reused 0 +Receiving objects: 100% (143/143), 85.57 KiB | 2.09 MiB/s, done. +Resolving deltas: 100% (7/7), done. +Available targets: + + aws/install Install aws cli bundle +``` + diff --git a/docs/extensions.md b/docs/extensions.md index 93de2d0ef..c21645ac3 100644 --- a/docs/extensions.md +++ b/docs/extensions.md @@ -1,6 +1,8 @@ + ## Extending `build-harness` with targets from another repo It is possible to extend the `build-harness` with targets and entire modules of your own, without having to fork or modify `build-harness` itself. This might be useful if, for example, you wanted to maintain some tooling that was specific to your environment that didn't have enough general applicability to be part of the main project. This makes it so you don't necessarily need to fork `build-harness` itself - you can place a repo defined by the environment variable `BUILD_HARNESS_EXTENSIONS_PATH` (a filesystem peer of `build-harness` named `build-harness-extensions` by default) and populate it with tools in the same `Makefile` within `module` structure as `build-harness` has. Modules will be combined and available with a unified `make` command. + diff --git a/docs/targets.md b/docs/targets.md index c8e32d33a..9ba84410d 100644 --- a/docs/targets.md +++ b/docs/targets.md @@ -1,5 +1,6 @@ + ## Makefile Targets -``` +```text Available targets: aws/install Install aws cli bundle @@ -7,6 +8,7 @@ Available targets: bash/lint Lint all bash scripts chamber/install Install chamber chamber/shell Start a chamber shell with secrets exported to the environment + clean Clean build-harness codefresh/export DEPRECATED!!! Export codefresh additional envvars codefresh/notify/slack/build Send notification from codefresh to slack using "build" template codefresh/notify/slack/deploy Send notification from codefresh to slack using "deploy" template @@ -95,6 +97,7 @@ Available targets: help Help screen help/all Display help for all targets help/short This help short screen + init Init build-harness jenkins/run-job-with-tag Run a Jenkins Job with $(TAG) make/lint Lint all makefiles packages/delete Delete packages @@ -113,13 +116,17 @@ Available targets: slack/notify/deploy Send notification to slack using "deploy" template template/build Create $OUT file by building it from $IN template file template/deps Install dependencies + terraform/bump-tf-12-min-version Rewrite versions.tf to bump modules with minimum core version of '0.12.x' to '>= 0.12.26' terraform/get-modules Ensure all modules can be fetched terraform/get-plugins Ensure all plugins can be fetched terraform/install Install terraform terraform/lint Lint check Terraform - terraform/upgrade-modules Upgrade all terraform module sources + terraform/loosen-constraints and convert "~>" constraints to ">=". + terraform/rewrite-required-providers Rewrite versions.tf to update existing configuration to add an explicit source attribute for each provider + terraform/upgrade-modules This target has not been upgraded to handle registry format terraform/validate Basic terraform sanity check travis/docker-login Login into docker hub travis/docker-tag-and-push Tag & Push according Travis environment variables ``` + diff --git a/modules/aws/Makefile b/modules/aws/Makefile index de9cb8026..f024d95bc 100644 --- a/modules/aws/Makefile +++ b/modules/aws/Makefile @@ -1,4 +1,4 @@ -export AWSCLI_VERSION ?= 1.11.185 +export AWSCLI_VERSION ?= 1.20.28 WITH_AWS ?= aws-vault exec $(AWS_PROFILE) -- diff --git a/modules/docs/Makefile b/modules/docs/Makefile index 6e63a4c4d..de808b147 100644 --- a/modules/docs/Makefile +++ b/modules/docs/Makefile @@ -6,13 +6,17 @@ docs/deps:: ## Update `docs/targets.md` from `make help` docs/targets.md: docs/deps @( \ + echo ""; \ echo "## Makefile Targets"; \ - echo '```'; \ + echo '```text'; \ $(SELF) --no-print-directory --quiet --silent $(DEFAULT_HELP_TARGET) | sed $$'s,\x1b\\[[0-9;]*[a-zA-Z],,g'; \ echo '```'; \ + echo ""; \ ) > $@ .PHONY : docs/terraform.md ## Update `docs/terraform.md` from `terraform-docs` docs/terraform.md: docs/deps packages/install/terraform-docs - @$(BUILD_HARNESS_PATH)/bin/terraform-docs.sh md . > $@ + @echo "" > $@ + @terraform-docs md . >> $@ + @echo "" >> $@ diff --git a/modules/github/Makefile.init b/modules/github/Makefile.init index f91586dfb..ef1a207f5 100644 --- a/modules/github/Makefile.init +++ b/modules/github/Makefile.init @@ -1,34 +1,39 @@ GITHUB_TEMPLATES = \ .github/CODEOWNERS \ - .github/workflows/auto-greet.yml \ - .github/workflows/auto-assign.yml \ - .github/workflows/auto-label.yml \ - .github/workflows/auto-readme.yml \ - .github/workflows/slash-command-dispatch.yml \ - .github/ISSUE_TEMPLATE/config.yml \ .github/PULL_REQUEST_TEMPLATE.md \ + .github/ISSUE_TEMPLATE/config.yml \ .github/ISSUE_TEMPLATE/feature_request.md \ .github/ISSUE_TEMPLATE/bug_report.md \ - .github/ISSUE_TEMPLATE/question.md + .github/ISSUE_TEMPLATE/question.md \ + .github/auto-release.yml \ + .github/workflows/auto-release.yml \ + .github/workflows/validate-codeowners.yml + +# Install extra configuration for terraform projects +GITHUB_TERRAFORM_TEMPLATES = .github/workflows/chatops.yml \ + .github/workflows/auto-context.yml \ + .github/workflows/auto-format.yml \ + .github/mergify.yml \ + .github/renovate.json + $(GITHUB_TEMPLATES): $(addprefix $(BUILD_HARNESS_PATH)/templates/, $(GITHUB_TEMPLATES)) mkdir -p $(dir $@) cp $(BUILD_HARNESS_PATH)/templates/$@ $@ git ls-files --error-unmatch $@ 2>/dev/null || git add $@ -.github/auto-assign.yml:: # do not overwrite config by default +$(GITHUB_TERRAFORM_TEMPLATES): $(addprefix $(BUILD_HARNESS_PATH)/templates/terraform/, $(GITHUB_TERRAFORM_TEMPLATES)) mkdir -p $(dir $@) - cp $(BUILD_HARNESS_PATH)/templates/$@ $@ + cp $(BUILD_HARNESS_PATH)/templates/terraform/$@ $@ git ls-files --error-unmatch $@ 2>/dev/null || git add $@ -.github/auto-label.yml:: # do not overwrite config by default - mkdir -p $(dir $@) - cp $(BUILD_HARNESS_PATH)/templates/$@ $@ - git ls-files --error-unmatch $@ 2>/dev/null || git add $@ +github/init/context.tf: + curl -o context.tf -fsSL https://raw.githubusercontent.com/cloudposse/terraform-null-label/master/exports/context.tf + git ls-files --error-unmatch context.tf 2>/dev/null || git add context.tf + @if [[ -d examples/complete ]]; then \ + cp -p context.tf examples/complete/context.tf ; \ + fi -.github/slash-command-dispatch.yml:: # do not overwrite config by default - mkdir -p $(dir $@) - cp $(BUILD_HARNESS_PATH)/templates/$@ $@ - git ls-files --error-unmatch $@ 2>/dev/null || git add $@ +.PHONY: $(GITHUB_TEMPLATES) $(GITHUB_TERRAFORM_TEMPLATES) github/init/context.tf -github/init: $(GITHUB_TEMPLATES) .github/auto-assign.yml .github/auto-label.yml .github/slash-command-dispatch.yml +github/init: $(GITHUB_TEMPLATES) $(if $(wildcard *.tf),$(GITHUB_TERRAFORM_TEMPLATES) github/init/context.tf) diff --git a/modules/go/Makefile b/modules/go/Makefile index 10e90a9f5..4902323dc 100644 --- a/modules/go/Makefile +++ b/modules/go/Makefile @@ -1 +1 @@ -GO:= $(shell which go 2>/dev/null) +GO := $(shell which go 2>/dev/null) diff --git a/modules/make/Makefile b/modules/make/Makefile index b598cdbdc..b5779aaa2 100644 --- a/modules/make/Makefile +++ b/modules/make/Makefile @@ -3,6 +3,9 @@ make/lint: @LINT=true \ find . -type f -name '*Makefile*' \ -type f '!' -name '*.example' \ - -type f '!' -path './vendor/*' -exec \ - /bin/sh -c 'echo "==> {}">/dev/stderr; make --include-dir=modules/ --just-print --dry-run --recon --no-print-directory --quiet --silent -f {}' \; > /dev/null - @$(SELF) bash/lint + -type f '!' -path './modules/packages/*' \ + -type f '!' -path './vendor/*' \ + -type f '!' -path './templates/Makefile.build-harness' \ + -type f '!' -path './Makefile' \ + -exec \ + /bin/sh -c 'echo "==> {}">/dev/stderr; make --file=modules/packages/Makefile --include-dir=modules/ --just-print --dry-run --recon --no-print-directory --quiet --silent -f {}' \; > /dev/null diff --git a/modules/packages/Makefile b/modules/packages/Makefile index bf9cdff22..8fb31f9cd 100644 --- a/modules/packages/Makefile +++ b/modules/packages/Makefile @@ -1,6 +1,7 @@ export INSTALL_PATH ?= $(BUILD_HARNESS_PATH)/vendor -export PACKAGES_VERSION ?= 0.95.0 +export PACKAGES_VERSION ?= master export PACKAGES_PATH ?= $(BUILD_HARNESS_PATH)/vendor/packages +export PACKAGES_PREFER_HOST ?= false ## Delete packages packages/delete: @@ -13,18 +14,23 @@ packages/reinstall: packages/delete packages/install ## Install packages packages/install: @if [ ! -d $(PACKAGES_PATH) ]; then \ - echo "Installing packages $(PACKAGES_VERSION)..."; \ - rm -rf $(PACKAGES_PATH); \ - $(GIT) clone -c advice.detachedHead=false --depth=1 -b $(PACKAGES_VERSION) https://github.com/cloudposse/packages.git $(PACKAGES_PATH); \ - rm -rf $(PACKAGES_PATH)/.git; \ + echo "* Installing packages $(PACKAGES_VERSION)..."; \ + rm -rf $(PACKAGES_PATH); \ + $(GIT) clone -c advice.detachedHead=false --depth=1 -b $(PACKAGES_VERSION) https://github.com/cloudposse/packages.git $(PACKAGES_PATH); \ + rm -rf $(PACKAGES_PATH)/.git; \ fi ## Install package (e.g. helm, helmfile, kubectl) -packages/install/%: packages/install - @if [ ! -x $(INSTALL_PATH)/$(subst packages/install/,,$@) ]; then \ - $(MAKE) -C $(PACKAGES_PATH)/install $(subst packages/install/,,$@); \ +packages/install/%: + @binary="$*"; \ + if [[ -x "$(INSTALL_PATH)/$$binary" ]]; then \ + echo "* Package $$binary already installed"; \ + elif [[ "$(PACKAGES_PREFER_HOST)" == "true" ]] && installed=$$(command -v $* 2>/dev/null); then \ + echo Using "$*" from "$$installed" ; \ else \ - echo "Package $(subst packages/install/,,$@) already installed"; \ + $(MAKE) packages/install && \ + echo "* Installing $* to $(INSTALL_PATH)" && \ + $(MAKE) -C $(PACKAGES_PATH)/install "$*"; \ fi; ## Reinstall package (e.g. helm, helmfile, kubectl) diff --git a/modules/pre-commit/Makefile b/modules/pre-commit/Makefile new file mode 100644 index 000000000..891b6b25f --- /dev/null +++ b/modules/pre-commit/Makefile @@ -0,0 +1,5 @@ + +.PHONY: pre-commit/run + +pre-commit/run: + pre-commit run --all-files diff --git a/modules/readme/Makefile b/modules/readme/Makefile index 8371d6baf..4155d1347 100644 --- a/modules/readme/Makefile +++ b/modules/readme/Makefile @@ -28,7 +28,10 @@ readme/lint: @rm -f $(README_LINT) ## Create README.md by building it from README.yaml -readme/build: $(README_DEPS) +readme/build: readme/deps $(README_DEPS) @gomplate --file $(README_TEMPLATE_FILE) \ --out $(README_FILE) @echo "Generated $(README_FILE) from $(README_TEMPLATE_FILE) using data from $(README_TEMPLATE_YAML)" + +readme/generate-related-references: + @$(BUILD_HARNESS_PATH)/bin/generate_related_references.py diff --git a/modules/terraform/Makefile b/modules/terraform/Makefile index a159f3e7b..89e956efe 100644 --- a/modules/terraform/Makefile +++ b/modules/terraform/Makefile @@ -1,6 +1,6 @@ TMP ?= /tmp TERRAFORM ?= $(BUILD_HARNESS_PATH)/vendor/terraform -TERRAFORM_VERSION ?= 0.11.11 +TERRAFORM_VERSION ?= 0.12.26 TERRAFORM_URL ?= https://releases.hashicorp.com/terraform/$(TERRAFORM_VERSION)/terraform_$(TERRAFORM_VERSION)_$(OS)_$(BUILD_HARNESS_ARCH).zip ## Install terraform @@ -32,9 +32,89 @@ endif ## Lint check Terraform terraform/lint: +ifeq ($(OS), darwin) + @FAIL=`$(TERRAFORM) fmt -write=false | xargs -n 1 printf '\t- %s\n'`; \ + [ -z "$$FAIL" ] || (echo "Terraform configuration needs linting. Run '$(TERRAFORM) fmt'"; echo $$FAIL; exit 1) +else @FAIL=`$(TERRAFORM) fmt -write=false | xargs --no-run-if-empty -n 1 printf '\t- %s\n'`; \ [ -z "$$FAIL" ] || (echo "Terraform configuration needs linting. Run '$(TERRAFORM) fmt'"; echo $$FAIL; exit 1) +endif + +terraform/fmt: +ifeq ($(wildcard *.tf),) + @echo "* $@: No terraform files detected" +else + @{ command -v terraform-0.13 && \ + command -v terraform-0.14 && \ + command -v terraform-0.15 && \ + command -v terraform-1 \ + ; } >/dev/null || { echo "* $@ requires binaries named terraform-0.13, terraform-0.14, terraform-0.15, and terraform-1" && false; } + @ # vert exits non-zero if any of the versions are not acceptable, so `|| [[ -n "$VERSION" ]]` for a real error check + @TF012=0.12.29; \ + TF013=$$(terraform-0.13 version --json | jq -r .terraform_version); \ + TF014=$$(terraform-0.14 version --json | jq -r .terraform_version); \ + TF015=$$(terraform-0.15 version --json | jq -r .terraform_version); \ + TF1=$$(terraform-1 version --json | jq -r .terraform_version); \ + VERSION=$$(vert -s "$$(terraform-config-inspect --json . | jq -r '.required_core[]')" "$$TF012" "$$TF013" "$$TF014" "$$TF015" "$$TF1" | head -1) || [[ -n "$$VERSION" ]]; \ + VERSION=$${VERSION:0:4}; \ + [[ $$VERSION =~ ^1 ]] && VERSION=1 || true; \ + terraform-$${VERSION} fmt -recursive . +endif ## Upgrade all terraform module sources -terraform/upgrade-modules: packages/install/json2hcl - @$(BUILD_HARNESS_PATH)/bin/upgrade_terraform_modules.sh all +## This target has not been upgraded to handle registry format +terraform/upgrade-modules: + @echo "* $@ has been disabled because it has not been updated to handle Terraform registry syntax" + # @$(BUILD_HARNESS_PATH)/bin/upgrade_terraform_modules.sh all + +# Rewrite the *.tf files to use registry notation for modules sources +terraform/rewrite-module-source: TERRAFORM = terraform-0.13 +terraform/rewrite-module-source: + @sed -i -E 's,\s*source\s+=\s+"git::https://github.com/([^/]+)/terraform-([^-]+)-(.+).git\?ref=(tags/)?([0-9.]+)", source = "\1/\3/\2"\n version = "\5",g' $$(find . -type f -not -name context.tf -name '*.tf') + @$(TERRAFORM) fmt . + @$(TERRAFORM) fmt examples/complete + +terraform/rewrite-readme-source: TERRAFORM = terraform-0.13 +terraform/rewrite-readme-source: + @sed -i -E 's,^(\s*)source\s+=\s+"git::https://github.com/([^/]+)/terraform-([^-]+)-(.+).git\?ref=(tags/)?master",\1source = "\2/\4/\3"\n\1# Cloud Posse recommends pinning every module to a specific version\n\1# version = "x.x.x",g' README.yaml + +## Rewrite versions.tf to remove upper bound for terraform core version constraint (like this ">= 0.12.0, < 0.14.0") +## and convert "~>" constraints to ">=". +terraform/loosen-constraints: TERRAFORM = terraform-0.13 +terraform/loosen-constraints: + @for v in $$(find . -type f -name 'versions.tf'); do \ + DIR="$$(dirname $$v)"; \ + sed -i -E 's,\s\=\s*\"(.*[0-9.]+)(\,\s*<.*)", = "\1",g' "$$v" ; \ + sed -i -E 's,=\s*"\s*~>,= ">=,g' "$$v" ; \ + $(TERRAFORM) fmt "$$DIR" ; \ + done + +## Rewrite versions.tf to bump modules with minimum core version of '0.12.x' to '>= 0.12.26' +terraform/bump-tf-12-min-version: TERRAFORM = terraform-0.13 +terraform/bump-tf-12-min-version: terraform/loosen-constraints + @for v in $$(find . -type f -name 'versions.tf'); do \ + DIR="$$(dirname $$v)"; \ + PIN="$$(terraform-config-inspect --json "$$DIR" | jq -r '.required_core[]')" ; \ + if vert "$$PIN" 0.12.25 >/dev/null; then \ + echo updating "$$v" ; \ + sed -i -E 's,required_version\s*\=\s*\"(\~>|>\=)\s?(0\.12(\.\d+)?)\",required_version = ">= 0.12.26",g' "$$v" ; \ + $(TERRAFORM) fmt $$DIR ; \ + else \ + echo "$$v" already pinned to "$$PIN" ; \ + fi; \ + done + +## Rewrite versions.tf to update existing configuration to add an explicit source attribute for each provider +terraform/rewrite-required-providers: TERRAFORM = terraform-0.13 +terraform/rewrite-required-providers: terraform/loosen-constraints + @for v in $$(find . -type f -name 'versions.tf'); do \ + DIR="$$(dirname $$v)" ; \ + KEEP_VERSION=$$(grep -Eo 'required_version\s*=\s*"(.*)"' "$$v" | cut -d '"' -f2) ; \ + echo upgrading $$v ; \ + terraform-0.13 0.13upgrade -yes "$$DIR" >/dev/null ; \ + sed -i -E 's,(required_version\s*\=\s*\")(.*)(\"),\1'"$$KEEP_VERSION"'\3,g' "$$v" ; \ + done + +terraform/v14-rewrite: TERRAFORM = terraform-0.13 +terraform/v14-rewrite: terraform/loosen-constraints terraform/bump-tf-12-min-version terraform/rewrite-required-providers terraform/rewrite-module-source terraform/rewrite-readme-source + @{ [[ "$(TERRAFORM_FORCE_README)" != "true" ]] && git diff --no-patch --exit-code README.yaml; } || $(MAKE) readme diff --git a/templates/.github/CODEOWNERS b/templates/.github/CODEOWNERS index 41c1baad5..6f64b5a33 100644 --- a/templates/.github/CODEOWNERS +++ b/templates/.github/CODEOWNERS @@ -1,4 +1,25 @@ # Use this file to define individuals or teams that are responsible for code in a repository. # Read more:
+
+
+template: |
+ $CHANGES
+
+replacers:
+# Remove irrelevant information from Renovate bot
+- search: '/(?<=---\s+)+^#.*(Renovate configuration|Configuration)(?:.|\n)*?This PR has been generated .*/gm'
+ replace: ''
+# Remove Renovate bot banner image
+- search: '/\[!\[[^\]]*Renovate\][^\]]*\](\([^)]*\))?\s*\n+/gm'
+ replace: ''
diff --git a/templates/.github/workflows/auto-assign.yml b/templates/.github/workflows/auto-assign.yml
deleted file mode 100644
index dba9e4f6f..000000000
--- a/templates/.github/workflows/auto-assign.yml
+++ /dev/null
@@ -1,13 +0,0 @@
-name: auto-assign
-on:
- pull_request:
- types: [opened, reopened]
-
-jobs:
- reviewers:
- runs-on: ubuntu-latest
- steps:
- - uses: cloudposse/actions/github/auto-assign@0.5.0
- with:
- repo-token: "${{ secrets.GITHUB_TOKEN }}"
- configuration-path: '.github/auto-assign.yml'
diff --git a/templates/.github/workflows/auto-greet.yml b/templates/.github/workflows/auto-greet.yml
deleted file mode 100644
index 284bd70d9..000000000
--- a/templates/.github/workflows/auto-greet.yml
+++ /dev/null
@@ -1,23 +0,0 @@
-name: auto-greet
-
-on: [pull_request, issues]
-
-jobs:
- comment:
- runs-on: ubuntu-latest
- steps:
- - uses: actions/first-interaction@v1
- with:
- repo-token: ${{ secrets.GITHUB_TOKEN }}
- issue-message: |-
- Thank you for reporting the issue! If you haven't already [joined our slack community](https://slack.sweetops.com), then we invite you to do so.
-
- This is a great place to get help and ask questions from our AMAZING community.
-
- [](https://slack.cloudposse.com)
- pr-message: |-
- Thank you for submitting this PR! If you haven't already [joined our slack community](https://slack.sweetops.com), then we invite you to do so.
-
- We receive an overwhelming number of contributions. By joining our slack, we'll be able to review your PR faster.
-
- [](https://slack.cloudposse.com)
diff --git a/templates/.github/workflows/auto-label.yml b/templates/.github/workflows/auto-label.yml
deleted file mode 100644
index 6c8047e2e..000000000
--- a/templates/.github/workflows/auto-label.yml
+++ /dev/null
@@ -1,11 +0,0 @@
-name: auto-label
-on: [pull_request]
-
-jobs:
- components:
- runs-on: ubuntu-latest
- steps:
- - uses: actions/labeler@v2.1.0
- with:
- repo-token: "${{ secrets.GITHUB_TOKEN }}"
- configuration-path: '.github/auto-label.yml'
diff --git a/templates/.github/workflows/auto-readme.yml b/templates/.github/workflows/auto-readme.yml
deleted file mode 100644
index 48bccff66..000000000
--- a/templates/.github/workflows/auto-readme.yml
+++ /dev/null
@@ -1,43 +0,0 @@
-name: 'auto-readme'
-
-on:
- # Open a PR to update README on merge to master (as necessary)
- push:
- branches:
- - master
-
- # Open a PR to update README for all PRs (as necessary)
- pull_request:
- types: [opened, synchronize, reopened]
-
- # Update README nightly
- schedule:
- - cron: '0 0 * * *'
-
-jobs:
- update:
- runs-on: ubuntu-latest
- steps:
- - name: checkout
- uses: actions/checkout@v1
- - name: readme/update
- shell: bash
- env:
- GITHUB_TOKEN: "${{ secrets.GITHUB_BOT_TOKEN }}"
- run: |
- make init
- make readme/deps
- make readme
- - name: Create Pull Request
- uses: cloudposse/actions/github/create-pull-request@0.4.0
- with:
- labels: automated pr
- env:
- GITHUB_TOKEN: ${{ secrets.GITHUB_BOT_TOKEN }}
- COMMIT_MESSAGE: Update README.md
- PULL_REQUEST_TITLE: Automatic Update of README.md
- PULL_REQUEST_BODY: |-
- This is an auto-generated PR which updates the `README.md` from the `README.yaml`
- using the [`cloudposse/build-harness`](https://github.com/cloudposse/build-harness).
- PULL_REQUEST_BRANCH: github-actions/auto-readme
- BRANCH_SUFFIX: none
diff --git a/templates/.github/workflows/auto-release.yml b/templates/.github/workflows/auto-release.yml
new file mode 100644
index 000000000..3a38fae08
--- /dev/null
+++ b/templates/.github/workflows/auto-release.yml
@@ -0,0 +1,26 @@
+name: auto-release
+
+on:
+ push:
+ branches:
+ - main
+ - master
+ - production
+
+jobs:
+ publish:
+ runs-on: ubuntu-latest
+ steps:
+ # Get PR from merged commit to master
+ - uses: actions-ecosystem/action-get-merged-pull-request@v1
+ id: get-merged-pull-request
+ with:
+ github_token: ${{ secrets.PUBLIC_REPO_ACCESS_TOKEN }}
+ # Drafts your next Release notes as Pull Requests are merged into "main"
+ - uses: release-drafter/release-drafter@v5
+ with:
+ publish: ${{ !contains(steps.get-merged-pull-request.outputs.labels, 'no-release') }}
+ prerelease: false
+ config-name: auto-release.yml
+ env:
+ GITHUB_TOKEN: ${{ secrets.PUBLIC_REPO_ACCESS_TOKEN }}
diff --git a/templates/.github/workflows/slash-command-dispatch.yml b/templates/.github/workflows/slash-command-dispatch.yml
deleted file mode 100644
index ebcef28c7..000000000
--- a/templates/.github/workflows/slash-command-dispatch.yml
+++ /dev/null
@@ -1,20 +0,0 @@
-name: Slash Command Dispatch
-on:
- issue_comment:
- types: [created]
-
-jobs:
- slashCommandDispatch:
- runs-on: ubuntu-latest
- steps:
- - uses: actions/checkout@v2
-
- - name: Slash Command Dispatch
- uses: cloudposse/actions/github/slash-command-dispatch@0.9.0
- with:
- token: ${{ secrets.GITHUB_BOT_TOKEN }}
- reaction-token: ${{ secrets.GITHUB_TOKEN }}
- repository: cloudposse/actions
- commands: rebuild-readme
- permission: none
- issue-type: pull-request
diff --git a/templates/.github/workflows/validate-codeowners.yml b/templates/.github/workflows/validate-codeowners.yml
new file mode 100644
index 000000000..c5193b625
--- /dev/null
+++ b/templates/.github/workflows/validate-codeowners.yml
@@ -0,0 +1,27 @@
+name: Validate Codeowners
+on:
+ workflow_dispatch:
+
+ pull_request:
+
+jobs:
+ validate-codeowners:
+ runs-on: ubuntu-latest
+ steps:
+ - name: "Checkout source code at current commit"
+ uses: actions/checkout@v2
+ - uses: mszostok/codeowners-validator@v0.5.0
+ if: github.event.pull_request.head.repo.full_name == github.repository
+ name: "Full check of CODEOWNERS"
+ with:
+ # For now, remove "files" check to allow CODEOWNERS to specify non-existent
+ # files so we can use the same CODEOWNERS file for Terraform and non-Terraform repos
+ # checks: "files,syntax,owners,duppatterns"
+ checks: "syntax,owners,duppatterns"
+ # GitHub access token is required only if the `owners` check is enabled
+ github_access_token: "${{ secrets.PUBLIC_REPO_ACCESS_TOKEN }}"
+ - uses: mszostok/codeowners-validator@v0.5.0
+ if: github.event.pull_request.head.repo.full_name != github.repository
+ name: "Syntax check of CODEOWNERS"
+ with:
+ checks: "syntax,duppatterns"
diff --git a/templates/Makefile.build-harness b/templates/Makefile.build-harness
index 5834360f3..04a6cf506 100644
--- a/templates/Makefile.build-harness
+++ b/templates/Makefile.build-harness
@@ -1,19 +1,173 @@
+#
+# This is a shim installed automatically by the build-harness
+# https://github.com/cloudposse/build-harness
+#
+
+# templates/Makefile.build-harness includes this Makefile
+# and this Makefile includes templates/Makefile.build-harness
+# to support different modes of invocation. Use a guard variable
+# to prevent infinite recursive includes
+ifeq ($(BUILD_HARNESS_TEMPLATES_MAKEFILE_GUARD),)
+BUILD_HARNESS_TEMPLATES_MAKEFILE_GUARD := included
+
export SHELL = /bin/bash
+export PWD = $(shell pwd)
export BUILD_HARNESS_ORG ?= cloudposse
export BUILD_HARNESS_PROJECT ?= build-harness
+export BUILD_HARNESS_DOCKER_IMAGE ?= $(BUILD_HARNESS_ORG)/$(BUILD_HARNESS_PROJECT)
export BUILD_HARNESS_BRANCH ?= master
-export BUILD_HARNESS_PATH ?= $(shell until [ -d "$(BUILD_HARNESS_PROJECT)" ] || [ "`pwd`" == '/' ]; do cd ..; done; pwd)/$(BUILD_HARNESS_PROJECT)
--include $(BUILD_HARNESS_PATH)/Makefile
+export BUILD_HARNESS_CLONE_URL ?= https://github.com/$(BUILD_HARNESS_ORG)/$(BUILD_HARNESS_PROJECT).git
+
+# Resolves BUILD_HARNESS_PATH to BUILD_HARNESS_PATH_LOCAL when BUILD_HARNESS_PATH does not exist
+BUILD_HARNESS_PATH ?= $(shell until [ -d "$(BUILD_HARNESS_PROJECT)" ] || [ "`pwd`" == '/' ]; do cd ..; done; pwd)/$(BUILD_HARNESS_PROJECT)
+BUILD_HARNESS_PATH_LOCAL := $(PWD)/$(BUILD_HARNESS_PROJECT)
+export BUILD_HARNESS_PATH := $(or $(wildcard $(BUILD_HARNESS_PATH)),$(BUILD_HARNESS_PATH_LOCAL))
+# It is kind of expensive to figure out the Docker SHA tag, so we just define the command here, and only call it when needed
+# With the ":=" syntax, it stores the current value of BUILD_HARNESS_PATH, so this has to come after that has been set with ":="
+export BUILD_HARNESS_DOCKER_SHA_TAG_CMD := git -C "$(BUILD_HARNESS_PATH)" log -n 1 --format=sha-%h 2>/dev/null || echo latest
+
+# Toggles the auto-init feature
+BUILD_HARNESS_AUTO_INIT ?= false
+
+# Macro to clone/install BUILD_HARNESS_PROJECT
+define harness_install
+curl --retry 5 --fail --silent --retry-delay 1 \
+ https://raw.githubusercontent.com/$(BUILD_HARNESS_ORG)/$(BUILD_HARNESS_PROJECT)/$(BUILD_HARNESS_BRANCH)/bin/install.sh | \
+ bash -s "$(BUILD_HARNESS_ORG)" "$(BUILD_HARNESS_PROJECT)" "$(BUILD_HARNESS_BRANCH)"
+endef
+
+# Macro to auto-init the BUILD_HARNESS_PROJECT with the `include` directive
+# Tests if BUILD_HARNESS_PROJECT does not yet exist, or if it does exist but the
+# checkout does not match BUILD_HARNESS_BRANCH
+define harness_auto_init
+if [[ \
+ -f "/build-harness/Makefile" || -f "/$(BUILD_HARNESS_PROJECT)/Makefile" \
+]]; then \
+ echo "[.build-harness]: In $(BUILD_HARNESS_PROJECT) docker container, skipping auto-init" ;\
+elif [[ \
+ grep -q docker /proc/1/cgroup 2>/dev/null \
+]]; then \
+ echo "[.build-harness]: In unknown docker container, skipping auto-init" ;\
+elif [[ \
+ "$(BUILD_HARNESS_PATH)" != "$(BUILD_HARNESS_PATH_LOCAL)" && \
+ -f "$(BUILD_HARNESS_PATH)/Makefile" \
+]]; then \
+ echo "[.build-harness]: Using external $(BUILD_HARNESS_PATH), skipping auto-init" ;\
+elif [[ \
+ "$(BUILD_HARNESS_PATH)" == "$(BUILD_HARNESS_PATH_LOCAL)" && \
+ -f "$(BUILD_HARNESS_PATH)/Makefile" && \
+ "$$(git -C '$(BUILD_HARNESS_PATH_LOCAL)' ls-remote '$(BUILD_HARNESS_CLONE_URL)' '$(BUILD_HARNESS_BRANCH)' | cut -f1)" == "$$(git -C '$(BUILD_HARNESS_PATH_LOCAL)' rev-parse HEAD)" \
+]]; then \
+ echo "[.build-harness]: Clone of $(BUILD_HARNESS_PROJECT) is up-to-date, skipping auto-init" ;\
+else \
+ $(harness_install) ;\
+fi
+endef
+
+-include $(if $(findstring true,$(BUILD_HARNESS_AUTO_INIT)),$(shell $(harness_auto_init) >&2)) $(BUILD_HARNESS_PATH)/Makefile
.PHONY : init
## Init build-harness
init::
- @curl --retry 5 --fail --silent --retry-delay 1 https://raw.githubusercontent.com/$(BUILD_HARNESS_ORG)/$(BUILD_HARNESS_PROJECT)/$(BUILD_HARNESS_BRANCH)/bin/install.sh | \
- bash -s "$(BUILD_HARNESS_ORG)" "$(BUILD_HARNESS_PROJECT)" "$(BUILD_HARNESS_BRANCH)"
+ @ $(harness_install)
.PHONY : clean
## Clean build-harness
clean::
@[ "$(BUILD_HARNESS_PATH)" == '/' ] || \
- [ "$(BUILD_HARNESS_PATH)" == '.' ] || \
- echo rm -rf $(BUILD_HARNESS_PATH)
+ [ "$(BUILD_HARNESS_PATH)" == '.' ] || \
+ [ "$(BUILD_HARNESS_PATH)" == '/$(BUILD_HARNESS_PROJECT)' ] || \
+ echo rm -rf $(BUILD_HARNESS_PATH)
+
+.PHONY: build-harness/shell builder build-harness/shell/pull builder/pull builder/build
+
+build-harness/shell/pull builder/pull builder/build: BUILD_HARNESS_DOCKER_SHA_TAG ?= $(shell $(BUILD_HARNESS_DOCKER_SHA_TAG_CMD))
+build-harness/shell/pull builder/pull:
+ docker pull $(BUILD_HARNESS_DOCKER_IMAGE):$(BUILD_HARNESS_DOCKER_SHA_TAG)
+ @[[ "$(BUILD_HARNESS_DOCKER_SHA_TAG)" == "latest" ]] || docker pull $(BUILD_HARNESS_DOCKER_IMAGE):latest
+
+builder/build: export DOCKER_IMAGE_NAME = $(BUILD_HARNESS_DOCKER_IMAGE):$(BUILD_HARNESS_DOCKER_SHA_TAG)
+builder/build:
+ @$(MAKE) --no-print-directory docker/build
+
+DEFAULT_DOCKER_ENVS := AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN TERM AWS_PROFILE AWS_REGION \
+ AWS_DEFAULT_PROFILE AWS_DEFAULT_REGION
+EXTRA_DOCKER_ENVS ?= AWS_CONFIG_FILE AWS_SHARED_CREDENTIALS_FILE
+DOCKER_ENVS ?= $(DEFAULT_DOCKER_ENVS) $(EXTRA_DOCKER_ENVS)
+
+## Start a shell inside of the `build-harness` docker container with `make build-harness/shell` or `make builder`
+## Run `make` targets inside the build-harness shell by setting `TARGETS` or `TARGET`, e.g.
+## make builder TARGETS="github/init readme"
+build-harness/shell builder tester: MOUNT_HOME ?= $(shell [ -d "$$HOME" ] && printf -- "-e HOME -v \"%s\":\"%s\"" "$$HOME" "$$HOME")
+build-harness/shell builder tester: TARGETS ?= $(TARGET)
+build-harness/shell builder tester: ARGS := $(if $(TARGETS),$(TARGETS),-l || true)
+build-harness/shell builder tester: ENTRYPOINT := $(if $(TARGETS),/usr/bin/make,/bin/bash)
+build-harness/shell builder pr/pre-commit: RUNNER_DOCKER_TAG ?= $(shell $(BUILD_HARNESS_DOCKER_SHA_TAG_CMD))
+build-harness/shell builder pr/pre-commit: RUNNER_DOCKER_IMAGE ?= $(BUILD_HARNESS_DOCKER_IMAGE)
+build-harness/shell builder: build-harness/runner
+ @exit 0
+
+.PHONY: build-harness/shell-slim builder-slim pr/auto-format pr/auto-format/host pr/readme pr/readme/host pr/pre-commit tf14-upgrade
+
+build-harness/shell-slim builder-slim pr/auto-format pr/readme tf14-upgrade: RUNNER_DOCKER_IMAGE ?= $(BUILD_HARNESS_DOCKER_IMAGE)
+
+build-harness/shell-slim builder-slim tf14-upgrade pr/auto-format pr/readme: RUNNER_DOCKER_SHA_TAG ?= $(shell $(BUILD_HARNESS_DOCKER_SHA_TAG_CMD))
+build-harness/shell-slim builder-slim tf14-upgrade pr/auto-format pr/readme: RUNNER_DOCKER_TAG ?= \
+ $(shell docker inspect --type=image $(RUNNER_DOCKER_IMAGE):$(RUNNER_DOCKER_SHA_TAG) >/dev/null 2>&1 && \
+ echo "$(RUNNER_DOCKER_SHA_TAG) " || echo "slim-$(RUNNER_DOCKER_SHA_TAG)")
+
+build-harness/shell-slim builder-slim: TARGETS ?= $(TARGET)
+build-harness/shell-slim builder-slim: ARGS := $(if $(TARGETS),$(TARGETS),-l || true)
+build-harness/shell-slim builder-slim: ENTRYPOINT := $(if $(TARGETS),/usr/bin/make,/bin/bash)
+build-harness/shell-slim builder-slim: build-harness/runner
+
+pr/auto-format pr/readme pr/pre-commit tf14-upgrade : ENTRYPOINT := /usr/bin/make
+
+pr/auto-format pr/auto-format/host: ARGS := terraform/fmt readme
+pr/readme pr/readme/host: ARGS := readme/deps readme
+pr/auto-format pr/readme: build-harness/runner
+pr/auto-format/host pr/readme/host:
+ $(MAKE) $(ARGS)
+
+pr/pre-commit: ARGS := pre-commit/run
+pr/pre-commit: build-harness/runner
+
+tf14-upgrade: export TERRAFORM_FORCE_README := true
+tf14-upgrade: ARGS := github/init terraform/v14-rewrite
+tf14-upgrade: build-harness/runner
+
+.PHONY: tester tester/pull
+
+tester tester/pull: TEST_HARNESS_DOCKER_IMAGE ?= cloudposse/test-harness
+tester tester/pull: TEST_HARNESS_DOCKER_TAG ?= latest
+tester: RUNNER_DOCKER_IMAGE ?= $(TEST_HARNESS_DOCKER_IMAGE)
+tester: RUNNER_DOCKER_TAG ?= $(TEST_HARNESS_DOCKER_TAG)
+tester: build-harness/runner
+
+tester/pull:
+ docker pull $(TEST_HARNESS_DOCKER_IMAGE):$(TEST_HARNESS_DOCKER_TAG)
+
+
+.PHONY: build-harness/runner
+
+build-harness/runner:
+ $(info Starting $(RUNNER_DOCKER_IMAGE):$(RUNNER_DOCKER_TAG))
+ docker run --name build-harness \
+ --rm -it \
+ -e PACKAGES_PREFER_HOST=true \
+ $(addprefix -e ,$(DOCKER_ENVS)) \
+ $(MOUNT_HOME) \
+ -v $(CURDIR):/opt \
+ --workdir /opt \
+ --entrypoint $(ENTRYPOINT) \
+ $(RUNNER_DOCKER_IMAGE):$(RUNNER_DOCKER_TAG) $(ARGS)
+
+.PHONY: reset-owner
+reset-owner:
+ @if [[ -n $$(find . -xdev -user 0 -print) ]]; then \
+ printf "\n* To reset ownership on files, run:\n sudo find . -xdev -user 0 -exec chown $$USER {} \;\n\n" ; \
+ else \
+ printf "\n* No root-owned files found\n\n" ; \
+ fi
+
+endif
diff --git a/templates/README.md.gotmpl b/templates/README.md.gotmpl
index 4882eacb6..d4c431667 100644
--- a/templates/README.md.gotmpl
+++ b/templates/README.md.gotmpl
@@ -1,24 +1,35 @@
-
+{{- if $deprecated }}
+# (deprecated) {{(ds "config").name}}{{ if gt (len (ds "config").name) 23 }}{{ print "\n\n" }}{{ end }}
+{{- else }}
+# {{(ds "config").name}}{{ if gt (len (ds "config").name) 34 }}{{ print "\n\n" }}{{ end }}
+{{- end }}
+{{- if $deprecated }}[](#deprecated){{ end }}
+ {{- if has (ds "config") "badges" }}
+ {{- range $badge := (ds "config").badges }}
+ {{- printf " [](%s)" $badge.name $badge.image $badge.url }}
+ {{- end }}
+{{- end }}
+
+[![README Header][readme_header_img]][readme_header_link]
+[![Cloud Posse][logo]](https://cpco.io/homepage)
+
-{{- defineDatasource "config" .Env.README_YAML | regexp.Replace ".*" "" -}}
-{{- defineDatasource "includes" .Env.README_INCLUDES | regexp.Replace ".*" "" }}
-[![README Header][readme_header_img]][readme_header_link]
-
-[![Cloud Posse][logo]](https://cpco.io/homepage)
-
-# {{(ds "config").name}}{{ if gt (len (ds "config").name) 34 }}{{ print "\n\n" }}{{ end }}{{ if has (ds "config") "badges" }}{{- range $badge := (ds "config").badges -}}{{ printf " [](%s)" $badge.name $badge.image $badge.url }}{{ end }}{{ end }}
-
+-->
{{ if has (ds "config") "logo" }}
.logo }})
{{- end -}}
+{{- if $deprecated }}
+## Deprecated
+
+{{ if has (ds "config").deprecated "notice" }}
+ {{- (ds "config").deprecated.notice }}
+{{- else }}
+ This module is no longer actively maintained
+{{- end }}
+{{- if (file.Exists "main.tf") }}
+
+We literally have [*hundreds of other terraform modules*][terraform_modules] that are Open Source and well-maintained. Check them out!
+{{- end }}
+
{{ if has (ds "config") "description" }}
+### Historical Description
+
{{(ds "config").description }}
-{{ end }}
+{{- end }}
+{{- else }}
+{{- if has (ds "config") "description" }}
+{{ (ds "config").description }}
+
+{{- end }}
+{{- end }}
---
-This project is part of our comprehensive ["SweetOps"](https://cpco.io/sweetops) approach towards DevOps.
+This project {{ if $deprecated }}was{{ else }}is{{ end }} part of our comprehensive ["SweetOps"](https://cpco.io/sweetops) approach towards DevOps.
[$TITLE @$AUTHOR (#$NUMBER)
+ + $BODY +][share_email]
[][share_googleplus]
[][share_facebook]
@@ -86,10 +100,10 @@ It's 100% Open Source and licensed under the [Internet Systems Consortium](LICEN
It's 100% Open Source and licensed under the [GNU General Public License](LICENSE).
{{ end }}
-{{ if (file.Exists "main.tf") }}
-We literally have [*hundreds of terraform modules*][terraform_modules] that are Open Source and well-maintained. Check them out!
-{{end}}
-
+{{ if not $deprecated }}
+{{- if (file.Exists "main.tf") }}
+We literally have [*hundreds of terraform modules*][terraform_modules] that are Open Source and well-maintained. Check them out!
+{{- end }}
{{ if has (ds "config") "screenshots" }}
@@ -97,23 +111,57 @@ We literally have [*hundreds of terraform modules*][terraform_modules] that are
{{ range $screenshot := (ds "config").screenshots }}
{{ printf "\n*%s*" $screenshot.name $screenshot.url $screenshot.description }}{{ end }}
-{{ end }}
+{{- end }}
+{{- end }}
+
{{ if has (ds "config") "introduction" }}
## Introduction
-{{ (ds "config").introduction -}}
+{{ (ds "config").introduction }}
{{ end }}
+
+{{ if (file.Exists "main.tf") }}
+{{- $repo_encoded := (ds "config").github_repo | regexp.Replace "/" "%2F" -}}
+## Security & Compliance [
](https://bridgecrew.io/)
+
+Security scanning is graciously provided by Bridgecrew. Bridgecrew is the leading fully hosted, cloud-native solution providing continuous Terraform security and compliance.
+
+| Benchmark | Description |
+|--------|---------------|
+{{- if $deprecated }}
+| [](#deprecated) | This project is no longer being maintained |
+{{- end }}
+| [![Infrastructure Security]({{ printf "https://www.bridgecrew.cloud/badges/github/%s/general" (ds "config").github_repo}})]({{ printf "https://www.bridgecrew.cloud/link/badge?vcs=github&fullRepo=%s&benchmark=INFRASTRUCTURE+SECURITY" $repo_encoded }}) | Infrastructure Security Compliance |
+| [![CIS KUBERNETES]({{ printf "https://www.bridgecrew.cloud/badges/github/%s/cis_kubernetes" (ds "config").github_repo}})]({{ printf "https://www.bridgecrew.cloud/link/badge?vcs=github&fullRepo=%s&benchmark=CIS+KUBERNETES+V1.5" $repo_encoded }}) | Center for Internet Security, KUBERNETES Compliance |
+| [![CIS AWS]({{ printf "https://www.bridgecrew.cloud/badges/github/%s/cis_aws" (ds "config").github_repo}})]({{ printf "https://www.bridgecrew.cloud/link/badge?vcs=github&fullRepo=%s&benchmark=CIS+AWS+V1.2" $repo_encoded }}) | Center for Internet Security, AWS Compliance |
+| [![CIS AZURE]({{ printf "https://www.bridgecrew.cloud/badges/github/%s/cis_azure" (ds "config").github_repo}})]({{ printf "https://www.bridgecrew.cloud/link/badge?vcs=github&fullRepo=%s&benchmark=CIS+AZURE+V1.1" $repo_encoded }}) | Center for Internet Security, AZURE Compliance |
+| [![PCI-DSS]({{ printf "https://www.bridgecrew.cloud/badges/github/%s/pci" (ds "config").github_repo}})]({{ printf "https://www.bridgecrew.cloud/link/badge?vcs=github&fullRepo=%s&benchmark=PCI-DSS+V3.2" $repo_encoded }}) | Payment Card Industry Data Security Standards Compliance |
+| [![NIST-800-53]({{ printf "https://www.bridgecrew.cloud/badges/github/%s/nist" (ds "config").github_repo}})]({{ printf "https://www.bridgecrew.cloud/link/badge?vcs=github&fullRepo=%s&benchmark=NIST-800-53" $repo_encoded }}) | National Institute of Standards and Technology Compliance |
+| [![ISO27001]({{ printf "https://www.bridgecrew.cloud/badges/github/%s/iso" (ds "config").github_repo}})]({{ printf "https://www.bridgecrew.cloud/link/badge?vcs=github&fullRepo=%s&benchmark=ISO27001" $repo_encoded }}) | Information Security Management System, ISO/IEC 27001 Compliance |
+| [![SOC2]({{ printf "https://www.bridgecrew.cloud/badges/github/%s/soc2" (ds "config").github_repo}})]({{ printf "https://www.bridgecrew.cloud/link/badge?vcs=github&fullRepo=%s&benchmark=SOC2" $repo_encoded }})| Service Organization Control 2 Compliance |
+| [![CIS GCP]({{ printf "https://www.bridgecrew.cloud/badges/github/%s/cis_gcp" (ds "config").github_repo}})]({{ printf "https://www.bridgecrew.cloud/link/badge?vcs=github&fullRepo=%s&benchmark=CIS+GCP+V1.1" $repo_encoded }}) | Center for Internet Security, GCP Compliance |
+| [![HIPAA]({{ printf "https://www.bridgecrew.cloud/badges/github/%s/hipaa" (ds "config").github_repo}})]({{ printf "https://www.bridgecrew.cloud/link/badge?vcs=github&fullRepo=%s&benchmark=HIPAA" $repo_encoded }}) | Health Insurance Portability and Accountability Compliance |
+{{ end }}
+
{{ if has (ds "config") "usage" }}
## Usage
{{ if (file.Exists "main.tf") }}
-**IMPORTANT:** The `master` branch is used in `source` just as an example. In your code, do not pin to `master` because there may be breaking changes between releases.
-Instead pin to the release tag (e.g. `?ref=tags/x.y.z`) of one of our [latest releases]({{ printf "https://github.com/%s/releases" (ds "config").github_repo}}).
+**IMPORTANT:** We do not pin modules to versions in our examples because of the
+difficulty of keeping the versions in the documentation in sync with the latest released versions.
+We highly recommend that in your code you pin the version to the exact version you are
+using so that your infrastructure remains stable, and update versions in a
+systematic way so that they do not catch you by surprise.
+
+Also, because of a bug in the Terraform registry ([hashicorp/terraform#21417](https://github.com/hashicorp/terraform/issues/21417)),
+the registry shows many of our inputs as required when in fact they are optional.
+The table below correctly indicates which inputs are required.
{{end}}
{{ (ds "config").usage -}}
{{ end }}
+{{ if not $deprecated -}}
{{ if has (ds "config") "quickstart" -}}
## Quick Start
@@ -133,32 +181,34 @@ Instead pin to the release tag (e.g. `?ref=tags/x.y.z`) of one of our [latest re
{{- end }}
{{ if has (ds "config") "related" }}
-## Share the Love
+## Share the Love
-Like this project? Please give it a ★ on [our GitHub]({{ printf "https://github.com/%s" (ds "config").github_repo}})! (it helps us **a lot**)
+Like this project? Please give it a ★ on [our GitHub]({{ printf "https://github.com/%s" (ds "config").github_repo}})! (it helps us **a lot**)
Are you using this project or any of our other projects? Consider [leaving a testimonial][testimonial]. =)
-
+{{ end }}
+{{ if has (ds "config") "related" }}
## Related Projects
Check out these related projects.
{{ range $related := (ds "config").related }}
{{ printf "- [%s](%s) - %s" $related.name $related.url $related.description }}{{ end }}
-{{ end}}
+{{- end}}
+{{- end}}
{{ if has (ds "config") "references" }}
## References
-For additional context, refer to some of these links.
+For additional context, refer to some of these links.
{{ range $reference := (ds "config").references }}
{{ printf "- [%s](%s) - %s" $reference.name $reference.url $reference.description }}{{ end }}
{{ end}}
## Help
-**Got a question?** We got answers.
+**Got a question?** We got answers.
File a GitHub [issue]({{ printf "https://github.com/%s/issues" (ds "config").github_repo}}), send us an [email][email] or join our [Slack Community][slack].
@@ -167,7 +217,7 @@ File a GitHub [issue]({{ printf "https://github.com/%s/issues" (ds "config").git
## DevOps Accelerator for Startups
-We are a [**DevOps Accelerator**][commercial_support]. We'll help you build your cloud infrastructure from the ground up so you can own it. Then we'll show you how to operate it and stick around for as long as you need us.
+We are a [**DevOps Accelerator**][commercial_support]. We'll help you build your cloud infrastructure from the ground up so you can own it. Then we'll show you how to operate it and stick around for as long as you need us.
[][commercial_support]
@@ -190,16 +240,21 @@ We deliver 10x the value for a fraction of the cost of a full-time engineer. Our
Join our [Open Source Community][slack] on Slack. It's **FREE** for everyone! Our "SweetOps" community is where you get to talk with others who share a similar vision for how to rollout and manage infrastructure. This is the best place to talk shop, ask questions, solicit feedback, and work together as a community to build totally *sweet* infrastructure.
+## Discourse Forums
+
+Participate in our [Discourse Forums][discourse]. Here you'll find answers to commonly asked questions. Most questions will be related to the enormous number of projects we support on our GitHub. Come here to collaborate on answers, find solutions, and get ideas about the products and services we value. It only takes a minute to get started! Just sign in with SSO using your GitHub account.
+
## Newsletter
-Sign up for [our newsletter][newsletter] that covers everything on our technology radar. Receive updates on what we're up to on GitHub as well as awesome new projects we discover.
+Sign up for [our newsletter][newsletter] that covers everything on our technology radar. Receive updates on what we're up to on GitHub as well as awesome new projects we discover.
## Office Hours
-[Join us every Wednesday via Zoom][office_hours] for our weekly "Lunch & Learn" sessions. It's **FREE** for everyone!
+[Join us every Wednesday via Zoom][office_hours] for our weekly "Lunch & Learn" sessions. It's **FREE** for everyone!
[![zoom](https://img.cloudposse.com/fit-in/200x200/https://cloudposse.com/wp-content/uploads/2019/08/Powered-by-Zoom.png")][office_hours]
+{{ if not $deprecated -}}
## Contributing
### Bug Reports & Feature Requests
@@ -219,7 +274,7 @@ In general, PRs are welcome. We follow the typical "fork-and-pull" Git workflow.
5. Submit a **Pull Request** so that we can review your changes
**NOTE:** Be sure to merge the latest changes from "upstream" before making a pull request!
-
+{{ end }}
{{ if has (ds "config") "copyrights" }}
## Copyrights
@@ -234,33 +289,35 @@ Copyright © 2017-{{ time.Now.Year }} [Cloud Posse, LLC](https://cpco.io/copyrig
{{ end}}
{{ if eq (ds "config").license "APACHE2" }}
-## License
+## License
-[](https://opensource.org/licenses/Apache-2.0)
+[](https://opensource.org/licenses/Apache-2.0)
See [LICENSE](LICENSE) for full details.
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- https://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
+```text
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+```
{{ end }}
{{ if eq (ds "config").license "CC-BY-NC-SA-4.0" }}
-## License
+## License
-[](https://creativecommons.org/licenses/by-nc-sa/4.0/)
+[](https://creativecommons.org/licenses/by-nc-sa/4.0/)
@@ -280,10 +337,11 @@ Distribution of the work or derivative of the work in any standard (paper) book
{{ end }}
{{ if eq (ds "config").license "MIT" }}
-## License
+## License
[](https://opensource.org/licenses/MIT)
+```text
The MIT License (MIT)
Permission is hereby granted, free of charge, to any person obtaining a copy
@@ -305,13 +363,15 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
Source: [%s][%s_homepage] |" $contributor.name $contributor.github $contributor.github $contributor.name $contributor.github}}{{ end }} |{{- range $contributor := (ds "config").contributors -}}---|{{ end }} + {{ range $contributor := (ds "config").contributors -}} {{- if has $contributor "homepage" }} @@ -396,6 +461,7 @@ Check out [our other projects][github], [follow us on twitter][twitter], [apply [testimonial]: {{ printf $utm_link "https://cpco.io/leave-testimonial" "testimonial" }} [office_hours]: {{ printf $utm_link "https://cloudposse.com/office-hours" "office_hours" }} [newsletter]: {{ printf $utm_link "https://cpco.io/newsletter" "newsletter" }} + [discourse]: {{ printf $utm_link "https://ask.sweetops.com/" "discourse" }} [email]: {{ printf $utm_link "https://cpco.io/email" "email" }} [commercial_support]: {{ printf $utm_link "https://cpco.io/commercial-support" "commercial_support" }} [we_love_open_source]: {{ printf $utm_link "https://cpco.io/we-love-open-source" "we_love_open_source" }} diff --git a/templates/terraform/.github/mergify.yml b/templates/terraform/.github/mergify.yml new file mode 100644 index 000000000..ef15545ec --- /dev/null +++ b/templates/terraform/.github/mergify.yml @@ -0,0 +1,65 @@ +# https://docs.mergify.io/conditions.html +# https://docs.mergify.io/actions.html +pull_request_rules: +- name: "approve automated PRs that have passed checks" + conditions: + - "author~=^(cloudpossebot|renovate\\[bot\\])$" + - "base=master" + - "-closed" + - "head~=^(auto-update|renovate)/.*" + - "check-success=test/bats" + - "check-success=test/readme" + - "check-success=test/terratest" + - "check-success=validate-codeowners" + actions: + review: + type: "APPROVE" + bot_account: "cloudposse-mergebot" + message: "We've automatically approved this PR because the checks from the automated Pull Request have passed." + +- name: "merge automated PRs when approved and tests pass" + conditions: + - "author~=^(cloudpossebot|renovate\\[bot\\])$" + - "base=master" + - "-closed" + - "head~=^(auto-update|renovate)/.*" + - "check-success=test/bats" + - "check-success=test/readme" + - "check-success=test/terratest" + - "check-success=validate-codeowners" + - "#approved-reviews-by>=1" + - "#changes-requested-reviews-by=0" + - "#commented-reviews-by=0" + actions: + merge: + method: "squash" + +- name: "delete the head branch after merge" + conditions: + - "merged" + actions: + delete_head_branch: {} + +- name: "ask to resolve conflict" + conditions: + - "conflict" + - "-closed" + actions: + comment: + message: "This pull request is now in conflict. Could you fix it @{{author}}? 🙏" + +- name: "remove outdated reviews" + conditions: + - "base=master" + actions: + dismiss_reviews: + changes_requested: true + approved: true + message: "This Pull Request has been updated, so we're dismissing all reviews." + +- name: "close Pull Requests without files changed" + conditions: + - "#files=0" + actions: + close: + message: "This pull request has been automatically closed by Mergify because there are no longer any changes." diff --git a/templates/terraform/.github/renovate.json b/templates/terraform/.github/renovate.json new file mode 100644 index 000000000..ae4f0aa54 --- /dev/null +++ b/templates/terraform/.github/renovate.json @@ -0,0 +1,12 @@ +{ + "extends": [ + "config:base", + ":preserveSemverRanges" + ], + "labels": ["auto-update"], + "enabledManagers": ["terraform"], + "terraform": { + "ignorePaths": ["**/context.tf", "examples/**"] + } +} + diff --git a/templates/terraform/.github/workflows/auto-context.yml b/templates/terraform/.github/workflows/auto-context.yml new file mode 100644 index 000000000..ab979e0eb --- /dev/null +++ b/templates/terraform/.github/workflows/auto-context.yml @@ -0,0 +1,57 @@ +name: "auto-context" +on: + schedule: + # Update context.tf nightly + - cron: '0 3 * * *' + +jobs: + update: + if: github.event_name == 'schedule' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + + - name: Update context.tf + shell: bash + id: update + env: + GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}" + run: | + if [[ -f context.tf ]]; then + echo "Discovered existing context.tf! Fetching most recent version to see if there is an update." + curl -o context.tf -fsSL https://raw.githubusercontent.com/cloudposse/terraform-null-label/master/exports/context.tf + if git diff --no-patch --exit-code context.tf; then + echo "No changes detected! Exiting the job..." + else + echo "context.tf file has changed. Update examples and rebuild README.md." + make init + make github/init/context.tf + make readme/build + echo "::set-output name=create_pull_request::true" + fi + else + echo "This module has not yet been updated to support the context.tf pattern! Please update in order to support automatic updates." + fi + + - name: Create Pull Request + if: steps.update.outputs.create_pull_request == 'true' + uses: cloudposse/actions/github/create-pull-request@0.22.0 + with: + token: ${{ secrets.PUBLIC_REPO_ACCESS_TOKEN }} + committer: 'cloudpossebot <11232728+cloudpossebot@users.noreply.github.com>' + author: 'cloudpossebot <11232728+cloudpossebot@users.noreply.github.com>' + commit-message: Update context.tf from origin source + title: Update context.tf + body: |- + ## what + This is an auto-generated PR that updates the `context.tf` file to the latest version from `cloudposse/terraform-null-label` + + ## why + To support all the features of the `context` interface. + + branch: auto-update/context.tf + base: master + delete-branch: true + labels: | + auto-update + context diff --git a/templates/terraform/.github/workflows/auto-format.yml b/templates/terraform/.github/workflows/auto-format.yml new file mode 100644 index 000000000..375d0fd47 --- /dev/null +++ b/templates/terraform/.github/workflows/auto-format.yml @@ -0,0 +1,88 @@ +name: Auto Format +on: + pull_request_target: + types: [opened, synchronize] + +jobs: + auto-format: + runs-on: ubuntu-latest + container: cloudposse/build-harness:latest + steps: + # Checkout the pull request branch + # "An action in a workflow run can’t trigger a new workflow run. For example, if an action pushes code using + # the repository’s GITHUB_TOKEN, a new workflow will not run even when the repository contains + # a workflow configured to run when push events occur." + # However, using a personal access token will cause events to be triggered. + # We need that to ensure a status gets posted after the auto-format commit. + # We also want to trigger tests if the auto-format made no changes. + - uses: actions/checkout@v2 + if: github.event.pull_request.state == 'open' + name: Privileged Checkout + with: + token: ${{ secrets.PUBLIC_REPO_ACCESS_TOKEN }} + repository: ${{ github.event.pull_request.head.repo.full_name }} + # Check out the PR commit, not the merge commit + # Use `ref` instead of `sha` to enable pushing back to `ref` + ref: ${{ github.event.pull_request.head.ref }} + + # Do all the formatting stuff + - name: Auto Format + if: github.event.pull_request.state == 'open' + shell: bash + env: + GITHUB_TOKEN: "${{ secrets.PUBLIC_REPO_ACCESS_TOKEN }}" + run: make BUILD_HARNESS_PATH=/build-harness PACKAGES_PREFER_HOST=true -f /build-harness/templates/Makefile.build-harness pr/auto-format/host + + # Commit changes (if any) to the PR branch + - name: Commit changes to the PR branch + if: github.event.pull_request.state == 'open' + shell: bash + id: commit + env: + SENDER: ${{ github.event.sender.login }} + run: | + set -x + output=$(git diff --name-only) + + if [ -n "$output" ]; then + echo "Changes detected. Pushing to the PR branch" + git config --global user.name 'cloudpossebot' + git config --global user.email '11232728+cloudpossebot@users.noreply.github.com' + git add -A + git commit -m "Auto Format" + # Prevent looping by not pushing changes in response to changes from cloudpossebot + [[ $SENDER == "cloudpossebot" ]] || git push + # Set status to fail, because the push should trigger another status check, + # and we use success to indicate the checks are finished. + printf "::set-output name=%s::%s\n" "changed" "true" + exit 1 + else + printf "::set-output name=%s::%s\n" "changed" "false" + echo "No changes detected" + fi + + - name: Auto Test + uses: cloudposse/actions/github/repository-dispatch@0.22.0 + # match users by ID because logins (user names) are inconsistent, + # for example in the REST API Renovate Bot is `renovate[bot]` but + # in GraphQL it is just `renovate`, plus there is a non-bot + # user `renovate` with ID 1832810. + # Mergify bot: 37929162 + # Renovate bot: 29139614 + # Cloudpossebot: 11232728 + # Need to use space separators to prevent "21" from matching "112144" + if: > + contains(' 37929162 29139614 11232728 ', format(' {0} ', github.event.pull_request.user.id)) + && steps.commit.outputs.changed == 'false' && github.event.pull_request.state == 'open' + with: + token: ${{ secrets.PUBLIC_REPO_ACCESS_TOKEN }} + repository: cloudposse/actions + event-type: test-command + client-payload: |- + { "slash_command":{"args": {"unnamed": {"all": "all", "arg1": "all"}}}, + "pull_request": ${{ toJSON(github.event.pull_request) }}, + "github":{"payload":{"repository": ${{ toJSON(github.event.repository) }}, + "comment": {"id": ""} + } + } + } diff --git a/templates/terraform/.github/workflows/chatops.yml b/templates/terraform/.github/workflows/chatops.yml new file mode 100644 index 000000000..4ddc06749 --- /dev/null +++ b/templates/terraform/.github/workflows/chatops.yml @@ -0,0 +1,37 @@ +name: chatops +on: + issue_comment: + types: [created] + +jobs: + default: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + - name: "Handle common commands" + uses: cloudposse/actions/github/slash-command-dispatch@0.22.0 + with: + token: ${{ secrets.PUBLIC_REPO_ACCESS_TOKEN }} + reaction-token: ${{ secrets.GITHUB_TOKEN }} + repository: cloudposse/actions + commands: rebuild-readme, terraform-fmt + permission: triage + issue-type: pull-request + + test: + runs-on: ubuntu-latest + steps: + - name: "Checkout commit" + uses: actions/checkout@v2 + - name: "Run tests" + uses: cloudposse/actions/github/slash-command-dispatch@0.22.0 + with: + token: ${{ secrets.PUBLIC_REPO_ACCESS_TOKEN }} + reaction-token: ${{ secrets.GITHUB_TOKEN }} + repository: cloudposse/actions + commands: test + permission: triage + issue-type: pull-request + reactions: false + +