generated from ministryofjustice/template-documentation-site
-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Nd 468 review re configure aws vault page (#754)
* Updated teh runbook to reflect the current baseline * Updated and reviewed the documentation to configure the AWS vault
- Loading branch information
Showing
2 changed files
with
336 additions
and
18 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,27 +1,345 @@ | ||
| --- | ||
| owner_slack: "#nvvs-devops" | ||
| title: Creating a GitHub PAT | ||
| last_reviewed_on: 2024-05-31 | ||
| title: Infrastructure CI Migration to GitHub Actions | ||
| last_reviewed_on: 2024-09-04 | ||
| review_in: 6 months | ||
| --- | ||
| # Generating a GitHub Personal Access Token | ||
| # Infrastructure CI Migration to GitHub Actions | ||
| This runbook outlines the steps to prepare a typical NVVS Terraform Infrastructure for migration to GitHUb Actions as per [ADR 011 - Use GitHub Actions for CI/CD](../documentation/adrs/011-use-github-actions-for-ci-cd.html). | ||
|
|
||
| We use a service GitHub account to create `personal access tokens`. | ||
| ## Benefits | ||
|
|
||
| ## Prerequisties | ||
| The procedure for migrating to GitHub Actions for CI will require some housekeeping and minor refactoring is necessary to enable the GitHub Action CI deployments. It will provide the following benefits. | ||
|
|
||
| - [1Password Enterprise](https://security-guidance.service.justice.gov.uk/password-storage-and-management-guide/#password-storage-and-management-guide) | ||
| - [oathtool](https://www.nongnu.org/oath-toolkit/oathtool.1.html) to generate a [one time password](https://en.wikipedia.org/wiki/One-time_password) | ||
| - Quicker deployments. | ||
| - Simple branch build and deploy. | ||
| - Clearer sequence display and log viewing for engineers to understand the context of deployment processes in action. | ||
| - Terraform version will be defined in one location(instead of Readme, buildspec or latest pulled in). | ||
| - Retrieval of TF_VAR inputs will be scripted and consistent (no need to assemble values form various sources indicated in example files and Readme). | ||
| - The above will enable quicker on-boarding of engineers. | ||
| - Linting and Security tools will enable overview of as yet undefined issues. | ||
|
|
||
| ## Accessing the service account details | ||
| > **Note:** | ||
| > Use an `incognito` browser session to generate token to be sure not to create in your own account. | ||
| ## Stages | ||
|
|
||
| Open the 1Password vault and search for [staff-infrastructure-moj](https://github.com/staff-infrastructure-moj). | ||
| Open a new incognito browser session and copy the user, password then create a one time password using the code in the secure notes on the GitHub challenge. | ||
| Initially the project repository will be prepared to run `terraform plan` on GitHub. When this is achieved the AWS CodeBuild pipeline can be disabled; prior to full migration to GitHub. The Terraform state and Dynamo DB state locking AWS resources would be managed via another Terraform Module or the current module would have option to remove the AWS Codepipeline resources. | ||
|
|
||
| ## Creating a PAT | ||
| > **Warning:** | ||
| > Treat your tokens like passwords and keep them secret. When working with the API, use tokens as environment variables instead of hardcoding them. | ||
| ### Overview | ||
|
|
||
| Once logged into the `staff-infrastructure-moj` account. Follow GitHub's latest guidance on creating a PAT [here](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token#creating-a-token). | ||
| - Makefile update | ||
| - README update | ||
| - Terraform Version | ||
| - Terraform Inputs (TF_VAR) | ||
| - GitHub Permissions & GitHub Secrets | ||
| - Test GitHub Action on branch | ||
| - Test Static Code Analysis | ||
|
|
||
| ### Makefile update | ||
|
|
||
| For consistency ensure the `Makefile` has all the commands here: | ||
|
|
||
| ```make | ||
| #!make | ||
| .DEFAULT_GOAL := help | ||
| SHELL := '/bin/bash' | ||
|
|
||
| CURRENT_TIME := `date "+%Y.%m.%d-%H.%M.%S"` | ||
| TERRAFORM_VERSION := `cat versions.tf 2> /dev/null | grep required_version | cut -d "\\"" -f 2 | cut -d " " -f 2` | ||
|
|
||
| LOCAL_IMAGE := ministryofjustice/nvvs/terraforms:latest | ||
| DOCKER_IMAGE := ghcr.io/ministryofjustice/nvvs/terraforms:latest | ||
|
|
||
| DOCKER_RUN := @docker run --rm \ | ||
| --env-file <(aws-vault exec $$AWS_PROFILE -- env | grep ^AWS_) \ | ||
| --env-file <(env | grep ^TF_VAR_) \ | ||
| --env-file <(env | grep ^ENV) \ | ||
| -e TFENV_TERRAFORM_VERSION=$(TERRAFORM_VERSION) \ | ||
| -v `pwd`:/data \ | ||
| --workdir /data \ | ||
| --platform linux/amd64 \ | ||
| $(DOCKER_IMAGE) | ||
|
|
||
| DOCKER_RUN_IT := @docker run --rm -it \ | ||
| --env-file <(aws-vault exec $$AWS_PROFILE -- env | grep ^AWS_) \ | ||
| --env-file <(env | grep ^TF_VAR_) \ | ||
| --env-file <(env | grep ^ENV) \ | ||
| -e TFENV_TERRAFORM_VERSION=$(TERRAFORM_VERSION) \ | ||
| -v `pwd`:/data \ | ||
| --workdir /data \ | ||
| --platform linux/amd64 \ | ||
| $(DOCKER_IMAGE) | ||
|
|
||
| export DOCKER_DEFAULT_PLATFORM=linux/amd64 | ||
|
|
||
| .PHONY: debug | ||
| debug: ## debug | ||
| @echo "debug" | ||
| $(info target is $@) | ||
|
|
||
| .PHONY: aws | ||
| aws: ## provide aws cli command as an arg e.g. (make aws AWSCLI_ARGUMENT="s3 ls") | ||
| $(DOCKER_RUN) /bin/bash -c "aws $$AWSCLI_ARGUMENT" | ||
|
|
||
| .PHONY: shell | ||
| shell: ## Run Docker container with interactive terminal | ||
| $(DOCKER_RUN_IT) /bin/bash | ||
|
|
||
| .PHONY: fmt | ||
| fmt: ## terraform fmt | ||
| $(DOCKER_RUN) terraform fmt --recursive | ||
|
|
||
| .PHONY: init | ||
| init: ## terraform init (make init ENV_ARGUMENT=pre-production) NOTE: Will also select the env's workspace. | ||
|
|
||
| ## INFO: Do not indent the conditional below, make stops with an error. | ||
| ifneq ("$(wildcard .env)","") | ||
| $(info Using config file ".env") | ||
| include .env | ||
| init: -init | ||
| else | ||
| $(info Config file ".env" does not exist.) | ||
| init: -init-gen-env | ||
| endif | ||
|
|
||
| .PHONY: -init-gen-env | ||
| -init-gen-env: | ||
| $(MAKE) gen-env | ||
| $(MAKE) -init | ||
|
|
||
| .PHONY: -init | ||
| -init: | ||
| $(DOCKER_RUN) terraform init --backend-config="key=terraform.$$ENV.state" | ||
| $(MAKE) workspace-select | ||
|
|
||
| .PHONY: init-upgrade | ||
| init-upgrade: ## terraform init -upgrade | ||
| $(DOCKER_RUN) terraform init -upgrade --backend-config="key=terraform.$$ENV.state" | ||
|
|
||
| .PHONY: import | ||
| import: ## terraform import e.g. (make import IMPORT_ARGUMENT="module.foo.bar some_resource") | ||
| $(DOCKER_RUN) terraform import $$IMPORT_ARGUMENT | ||
|
|
||
| .PHONY: rm | ||
| rm: ## terraform import e.g. (make rm RM_ARGUMENT="module.foo.bar") | ||
| $(DOCKER_RUN) terraform state rm $$RM_ARGUMENT | ||
|
|
||
| .PHONY: workspace-list | ||
| workspace-list: ## terraform workspace list | ||
| $(DOCKER_RUN) terraform workspace list | ||
|
|
||
| .PHONY: workspace-select | ||
| workspace-select: ## terraform workspace select | ||
| $(DOCKER_RUN) terraform workspace select $$ENV || \ | ||
| $(DOCKER_RUN) terraform workspace new $$ENV | ||
|
|
||
| .PHONY: validate | ||
| validate: ## terraform validate | ||
| $(DOCKER_RUN) terraform validate | ||
|
|
||
| .PHONY: plan-out | ||
| plan-out: ## terraform plan - output to timestamped file | ||
| $(DOCKER_RUN) terraform plan -no-color > $$ENV.$(CURRENT_TIME).tfplan | ||
|
|
||
| .PHONY: plan | ||
| plan: ## terraform plan | ||
| $(DOCKER_RUN) terraform plan | ||
|
|
||
| .PHONY: refresh | ||
| refresh: ## terraform refresh | ||
| $(DOCKER_RUN) terraform refresh | ||
|
|
||
| .PHONY: output | ||
| output: ## terraform output (make output OUTPUT_ARGUMENT='--raw dns_dhcp_vpc_id') | ||
| $(DOCKER_RUN) terraform output -no-color $$OUTPUT_ARGUMENT | ||
|
|
||
| .PHONY: output-bastion-rds-admin | ||
| output-bastion-rds-admin: ## terraform output (make output-bastion-rds-admin) | ||
| $(DOCKER_RUN) /bin/bash -c "terraform output -no-color -json rds_bastion | jq -r .admin[][]" | ||
|
|
||
| .PHONY: output-bastion-rds-server | ||
| output-bastion-rds-server: ## terraform output (make output-bastion-rds-server) | ||
| $(DOCKER_RUN) /bin/bash -c "terraform output -no-color -json rds_bastion | jq -r .server[][]" | ||
|
|
||
| .PHONY: output-bastion-rds-load_testing | ||
| output-bastion-rds-load_testing: ## terraform output (make output-bastion-rds-load_testing) | ||
| $(DOCKER_RUN) /bin/bash -c "terraform output -no-color -json rds_bastion | jq -r .load_testing[][]" | ||
|
|
||
| .PHONY: apply | ||
| apply: ## terraform apply | ||
| $(DOCKER_RUN_IT) terraform apply | ||
| $(DOCKER_RUN) /bin/bash -c "./scripts/publish_terraform_outputs.sh" | ||
|
|
||
| .PHONY: state-list | ||
| state-list: ## terraform state list | ||
| $(DOCKER_RUN) terraform state list | ||
|
|
||
| .PHONY: show | ||
| show: ## terraform show | ||
| $(DOCKER_RUN) terraform show -no-color | ||
|
|
||
| .PHONY: destroy | ||
| destroy: ## terraform destroy | ||
| $(DOCKER_RUN) terraform destroy | ||
|
|
||
| .PHONY: lock | ||
| lock: ## terraform providers lock (reset hashes after upgrades prior to commit) | ||
| rm .terraform.lock.hcl | ||
| $(DOCKER_RUN) terraform providers lock -platform=windows_amd64 -platform=darwin_amd64 -platform=linux_amd64 | ||
|
|
||
| .PHONY: clean | ||
| clean: ## clean terraform cached providers etc | ||
| rm -rf .terraform/ terraform.tfstate* .env | ||
|
|
||
| .PHONY: gen-env | ||
| gen-env: ## generate a ".env" file with the correct TF_VARS for the environment e.g. (make gen-env ENV_ARGUMENT=pre-production) | ||
| $(DOCKER_RUN) /bin/bash -c "./scripts/generate-env-file.sh $$ENV_ARGUMENT" | ||
|
|
||
| .PHONY: aws_describe_instances | ||
| aws_describe_instances: ## Use AWS CLI to describe EC2 instances - outputs a table with instance id, type, IP and name for current environment | ||
| $(DOCKER_RUN) /bin/bash -c "./scripts/aws_describe_instances.sh" | ||
|
|
||
| .PHONY: aws_ssm_start_session | ||
| aws_ssm_start_session: ## Use AWS CLI to start SSM session on an EC2 instance (make aws_ssm_start_session INSTANCE_ID=i-01d4de517c7336ff3) | ||
| $(DOCKER_RUN_IT) /bin/bash -c "./scripts/aws_ssm_start_session.sh $$INSTANCE_ID" | ||
|
|
||
| .PHONY: tfenv | ||
| tfenv: ## tfenv pin - terraform version from versions.tf | ||
| tfenv use $(cat versions.tf 2> /dev/null | grep required_version | cut -d "\"" -f 2 | cut -d " " -f 2) && tfenv pin | ||
|
|
||
| help: | ||
| @grep -h -E '^[a-zA-Z0-9_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}' | ||
| ``` | ||
|
|
||
| ### README update | ||
|
|
||
| Find the appropriate place to add the following snippet into the `README.md` or other documentation file. | ||
|
|
||
| <pre> | ||
| This file is no longer necessary. The necessary TF*VARS that are required from the SSM Parameter Store and used by Terraform are for local development and testing written to the `.env` file that the Makefile sources. The values are exported in the shell's environment as `TF_VAR*{variable_name}`. | ||
| Provided the following have been set in your shell's environment | ||
| ```shell | ||
| export AWS_PROFILE=mojo-shared-services-cli | ||
| ``` | ||
| You can run from the root of this project the following script. | ||
| ```shell | ||
| ./scripts/generate-env-file.sh [environment_name: development|pre-production|production] | ||
| ``` | ||
| An `.env` file will be produced for the environment, if you need to test or check a plan against another environment rerun the script. | ||
| ``` | ||
| </pre> | ||
|
|
||
|
|
||
| ### Terraform Version | ||
|
|
||
| Each Terraform project will have a `version.tf` file as per Hashcorp advised convention in the root of the project. It will have at a minimum a `terraform` block with the `required_version specified`. If the version has already been spefified in the `terraform` block in `main.tf` remove from there and add as illustrated. | ||
|
|
||
| ```terraform | ||
| terraform { | ||
| required_version = "1.1.8" | ||
| } | ||
| ``` | ||
|
|
||
| This convention enables easy retrieval of the Terraform version by automation scripts and for local use with the [`tfenv`](https://github.com/tfutils/tfenv) tool. The alias below will install the required Terraform version loaccly and pin it for the project. | ||
|
|
||
| ```shell | ||
| alias tfenvuse='tfenv use $(cat versions.tf 2> /dev/null | grep required_version | cut -d "\"" -f 2 | cut -d " " -f 2) && tfenv pin' | ||
| ``` | ||
|
|
||
|
|
||
|
|
||
| ### Terraform Inputs (TF_VAR) | ||
|
|
||
| Getting the correct values for `.env` files and `terraform.tfvars` files from the various locations is time consuming and doesn't allow a new engineer to get familiar with the tooling or project with any confidence. A script will gather the required values and produce an `.env` for the specified environment which will be exported into the shell's environment when a command is run via the `make` target (init, plan, apply etc). | ||
| The script will use the existing information to inform what is necessary, the `env` section of the `buildspec.yml` is the most reliable source of required variables. | ||
|
|
||
| Copy the scripts from [network-access-control-infrastructure/scripts](https://github.com/ministryofjustice/network-access-control-infrastructure/tree/main/scripts) to the current project's `scripts` directory (create if not already exists). | ||
|
|
||
| ```shell | ||
| mkdir -p scripts && \ | ||
| cp ../network-access-control-infrastructure/scripts/aws_ssm_get_parameters.sh scripts/ && \ | ||
| cp ../network-access-control-infrastructure/scripts/generate-env-file.sh scripts/ && \ | ||
| cp ../network-access-control-infrastructure/scripts/generate-github-env.sh scripts/ | ||
| ``` | ||
|
|
||
| The scripts `generate-env-file.sh` and `generate-github-env.sh` are similar. | ||
| The first is used locally the second within the GitHub Action Workflow file to generate an `.env` file or inject into the special `GITHUB_ENV` variable. | ||
|
|
||
| | Script | Use | Change | | ||
| |-------------------------------- |---------------------------- |---------------- | | ||
| | [aws_ssm_get_parameters.sh][get] | Retrieve SSM Params | most | | ||
| | [generate-env-file.sh][env] | locally generate an `.env` | lines 72 -> 88 | | ||
| | [generate-github-env.sh][hub] | GitHub Action Workflow | lines 11 -> 23 | | ||
|
|
||
| With reference to the `buildspec.yml` up the lines as indicated in the table with the values from the `env:variables:` yaml block in the `generate-*` script files. | ||
|
|
||
| The `aws_ssm_get_parameters.sh` needs updating with the values from the `env:parameter-store:` yaml block. The SSM Parameters need to be retrieved in blocks of ten. Due to variable names sometimes changing from parameter store to their use in Terraform a line for each need to be written in which they're mapped into an array for use in the other two scripts. | ||
| Note: start with one param, test then add and test. | ||
|
|
||
| ```shell | ||
| ./scripts/generate-env-file.sh [environment_name: development|pre-production|production] | ||
| ``` | ||
|
|
||
| Now to run the Terraform the Makefile will only require the `.env` to provide the variables. Test before proceeding. | ||
|
|
||
| ### GitHub Permissions & GitHub Secrets | ||
|
|
||
| The GitHub repository will likely need adding to the OIDC Provider and the AWS Role added as a GitHub secret. | ||
| `role-to-assume: ${{ secrets.AWS_ROLE_TO_ASSUME }}` | ||
| Checkout the repo [mojo-aws-github-oidc-provider](https://github.com/ministryofjustice/mojo-aws-github-oidc-provider) and add the name of the GitHub repository to the | ||
| [locals.tf](https://github.com/ministryofjustice/mojo-aws-github-oidc-provider/blob/main/locals.tf) | ||
|
|
||
| The code needs to be run from your local machine (it can't apply the code to it's self (yet). See that project's README file for full details. | ||
|
|
||
| ### Test GitHub Action on branch | ||
|
|
||
| Most project already have GitHub Actions Workflows although they don't require AWS credentials to run such as formatting and Dependabot actions; hence the `.github/workflows` directory will likely already exist. Add the workflow files from [network-access-control-infrastructure/.github/workflows](https://github.com/ministryofjustice/network-access-control-infrastructure/tree/227-update-makefile-readme-1/.github/workflows) to the current project's `scripts` directory (create if not already exists). | ||
|
|
||
| ```shell | ||
| mkdir -p .github/workflows && \ | ||
| cp ../network-access-control-infrastructure/.github/workflows/terraform-apply.yaml .github/workflows/ && \ | ||
| cp ../network-access-control-infrastructure/.github/workflows/terraform-static-analysis.yml .github/workflows/ | ||
| ``` | ||
|
|
||
| Those file don't need any changes, commit and push them. Currently they will only run `terraform plan` when there is a PR for the `main` branch or a push to a branch ending with `-GHTEST`. | ||
| To test you branch create a temporary branch from your current one and append `-GHTEST` | ||
|
|
||
| ```shell | ||
| git checkout -b {your_branch_name}-GHTEST | ||
| git push --set-upstream origin {your_branch_name}-GHTEST | ||
| ``` | ||
|
|
||
| Now you should see under the *Actions* tab these two workflows `https://github.com/ministryofjustice/{project_name}/actions`. | ||
| Check the results confirm they run. | ||
|
|
||
|
|
||
| ## Checklist | ||
|
|
||
| <ol> | ||
| <li> | ||
| <label>Makefile update</label> | ||
| <input type="checkbox" value="completed" name="status"> | ||
| </li> | ||
| <li> | ||
| <label>README update</label> | ||
| <input type="checkbox" value="completed" name="status"> | ||
| </li> | ||
| <li> | ||
| <label>Terraform Version</label> | ||
| <input type="checkbox" value="completed" name="status"> | ||
| </li> | ||
| <li> | ||
| <label>Terraform Inputs (TF_VAR)</label> | ||
| <input type="checkbox" value="completed" name="status"> | ||
| </li> | ||
| <li> | ||
| <label>GitHub Permissions & GitHub Secrets</label> | ||
| <input type="checkbox" value="completed" name="status"> | ||
| </li> | ||
| <li> | ||
| <label>Test GitHub Action on branch</label> | ||
| <input type="checkbox" value="completed" name="status"> | ||
| </li> | ||
| </ol> | ||
|
|
||
| [get]: https://github.com/ministryofjustice/network-access-control-infrastructure/blob/main/scripts/aws_ssm_get_parameters.sh | ||
| [env]: https://github.com/ministryofjustice/network-access-control-infrastructure/blob/main/scripts/generate-env-file.sh | ||
| [hub]: https://github.com/ministryofjustice/network-access-control-infrastructure/blob/main/scripts/generate-github-env.sh |