All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog and this project adheres to Semantic Versioning.
kafka_connect_cluster
now connects to Kafka via TLS
- Expose
BuildkiteAgentTags
variable inbuildkite_elastic_stack
- Added support for
on_demand_percentage
tobuildkite_elastic_stack
- Changed
buildkite_elastic_stack
module to use variables for template v6.10.0.
- Fixed cloudfront 500 error page to remove hardcoded PackManager
- Added AWS managed rules to WAF.
- Removed
snowflake_connector_config
module
It was re-written using a customer Terraform provider and inlined into the Data Platform project. It was created solely for the Platform team and should not be used elsewhere.
- Cloudfront module custom response errors. No longer serving custom 404, we will let the application process the response as it needs. Keep only custom errors for 500s.
- Additional maintenance page status codes
- Additional cloudfront module outputs
- Cloudfront maintenance mode
- Custom error responses
- Maintenance mode
- Default root object (for SPA's)
- Allows customizing default cache behavior
- Adds
waf
module - Fixes
cloudfront
module
- Adds
cloudfront
module with the option to setweb_acl_id
with WAF configuration - Generalizes pattern used on different systems for consistency
- Fixes configuration changes to bucket ACLs introduced by AWS on April, 2023
- Removed
launch_type
attribute fromecs_service
- Added support for ECS capacity provider. Defaults to
FARGATE
.
- Added
image_lifecycle_count
key to control ECR replication counts
- Added
kms_multi_region_key
to be used initially byhelm_secrets
module to encrypt/decrypt secrets using a KMS key that is highly available across primary and replica regions
- Added
access_log_bucket
andaccess_log_prefix
variables toecs_fargate_with_elb
& optionalaccess_logs
topublic_load_balancer
This allows access logs to be enabled for public load balancers.
snowflake_connector_config
exposes more variables for customization
kafka_connect_cluster
renamed thekafka_bootstrap_servers
tokafka_bootstrap_brokers
- Added
additional_envars
tokafka_connect_cluster
This allows creating environment variables to configure Kafka Connnect.
- Opinionated LaunchDarkly module that works in tandem with the Nulogy
feature_flagging
gem
buildkite_elastic_stack
now supports receiving aninstance_role_name
variable to explicitly set the CloudformationInstanceRoleName
needed to specify the name of the IAM role auto-created by the Cloudformation stack.
- Allow multiple Buildkite queues per account by assuring that agent token SSM name and Cloudformation stack names are unique
- Added
snowflake_connector_config
module to configure Kafka Connect Connector for Snowflake
- Added input parameter
kafka_connect__docker_image_name
tokafka_connect_cluster
module to allow different Kafka Connect Connectors to be run. This change is backwards compatible, as Debezium 1.6.2 is used as a default.
- ECS service fargate modules expose the task execution role ID so that additional policies can be attached
- ECS service fargate autoscaling module exposes the container name as an output
- ECS service fargate autoscaling module provides the container name template variable for the task definition JSON file
- ECS service autoscaling fargate modules are configured to work with service discovery by default
- ECS service autoscaling fargate modules are configured to
allow
enable_execute_command
- Removed service discovery configuration from CodeDeploy ECS module since it is not being used at the moment. It is not necessary for CodeDeploy based ECS tasks to communicate with other ECS services through service discovery.
- Added module
kafka_connect_cluster
for starting an isolated set of ECS tasks for each Kafka Connect service (e.g. Debezium for Message Bus in OpsCore NA)
- Added variable
subscription_events_ttl
todebezium_config
defaulting to seven days (to match Message Bus Kafka topic deletion policy). - Added variable
postgres_search_paths
todebezium_config
to modify the Postgressearch_path
when executing heartbeats.
- Removed the
heartbeat_query
variable fromdebezium_config
.
- Allow overriding the Postgres replication slot name when configuring Debezium
- Use the same
us-east-1
region for reaching the Logz.io listener
- Upgrade the CloudWatch Logz.io log shipping lambda module to Python 3.7
- ECS service discovery to
ecs_cluster
- ECS service fargate modules are configured to work with service discovery by default
- ECS service fargate modules are configured to allow
enable_execute_command
- Upgrade Buildkite CI Stack to 5.7.1
- Store Buildkite agent token in parameter store. A KMS key is required to encrypt the parameter store.
- More sensible defaults in
buildkite_elastic_stack
variables.tf
- Buildkite instances have no public IP by default
- Require Buildkite instances to use IMDSv2 for improved security usage of EC2 metadata service
- Added optional field to
ecs_fargate_with_elb
andecs_service_fargate_elb
to allow custom ECS service names
- Added
data_platform_database_user
module for creating a Postgres user for data platform.
- Changed module variables in
debezium_connection
to separate additionoal SQL statements for granting and revoking the user
- Added variable to run addtional SQL commands for
debezium_database_user
.
cloudfront_s3_origin
module that can be used to set up an AWS CloudFront distribution pointing to an S3 bucket as the origin. Useful for setting up HTTPS redirects.s3_redirect_bucket
module whose job is to provide a redirect to another URL.cloudfront_s3_redirect
module group that utilizes thecloudfront_s3_origin
ands3_redirect_bucket
modules to perform a HTTPS redirect to another URL.
ecs_service_fargate_codedeploy
module that uses AWS CodeDeploy for deployments.public_load_balancer_blue_green
module that has two target groups called Blue and Green which are used for AWS CodeDeploy for deployments.ecs_fargate_with_codedeploy
module group that uses AWS CodeDeploy for deployments and requires two target groups to be available called Blue and Green.
- Added the zone_id output to the ecs_fargate_with_elb module group
- Uses the
depends_on
meta argument formodules
on ECS service module to replace the existing workaround that was there
- Added
ssm:GetParametersByPath
permission to every role that was usingssm:GetParameter
cloudwatch_logzio
module also considers environment variables of the log shipping lambda when determining if it needs to apply any changes
cloudwatch_logzio
module does not specify any providers or versions so that it is compatible with TF 0.13cloudwatch_logzio
module takes the Logz.io API key as a parameter called:logzio__api_key
- Updated
debezium_connection
to insert heartbeats when running TF and when connecting to a database.
This should ensure heartbeats properly start for databases that have no traffic.
cloudwatch_logzio
module that can be used to send logs to Logz.io
- Updates
buildkite_elastic_stack
to be compatible with elastic stack v5
- Allow kafka_topics to work with TF 0.13
- Allow vpc_peering to work with TF 0.13
- Paramaterizes AssociatePublicIPAddress for the buildkite elastic stack module.
- Changes the default heartbeat SQL to use the renamed database columns.
- Added modules for apps to use to connect to the message bus as a
producer.
debezium_connection
anddeebziucm_database_user
.
- Added security groups variable to ecs_scheduled_task module
- Exposes the cluster ARN from the ecs_cluster module & module_group outputs
- Added variable
ecr_url
on the event_shovel module group to control which repository to pull EventShovel from. Defaults to CPI's shared internal repo
- No longer creates an ECR repo for EventShovel.
This was initially released in a branch as part of a bad merge. The tag has since been corrected. AK.
- Updates the VPC Peering to use the new version which updates the route tables and has a simpler API.
- Removes a second VPC peering module that appeared to be incomplete
- Add
kafka_topics
module so that Message Bus consumers can setup topics
- Updates the VPC Peering to use the new version which updates the route tables.
- Removes a second VPC peering module that appeared to be incomplete
- Add task_role_id output to ecs_fargate_with_elb module
- Rename iam_id output to task_role_id in ecs_service_fargate_elb
- Add container_name output to ecs_service_fargate module
- Migrated ecs_scheduled_task module to use fargate
- ecs_service_fargate_autoscaling, which is a version of the module that ignores desired_count changes so it can remain compatible with AWS autoscaling rules
- Renamed output in ecs_service_fargate from iam_id to task_role_id output
- ecs_service_fargate.iam_id output
- ecs_plb_platform module
- write access to param store for ecs service
- Selects most recent ACM certificate when there are more than one eligible, instead of crashing
- Marks the deployments using Datadog
- Resizes Airbrake panel on Datadog
- Added the Service Events panel
- buildkite_elastic_stack no longer takes the office ip and insteads allows traffic from the VPN
- Removed warning threshold as it has no difference in paging as the alert threshold.
- Added datadog module capable of creating dashboards and monitors.
- Added a data-only module for getting networking information about the Shared VPC
- Load Dadadog api key from parameter store
- Added Datadog env variable
DD_ENV
- Added Datadog version variable
DD_VERSION
- ECS service fargate task definition path
- Output for instance_role_name of buildkite stack
- Injects
DATADOG_SERVICE
env var into the app (feature flag for initializer)
- Add service name to Datadog agent tagging of a ECS Fargate task
- Improves Datadog agent tagging of ECS Fargate task
- Added
account_data
module
- Add option to enable Datadog agent sidecar/replica container for
the
ecs_fargate_with_elb
module group - Added Datadog agent container task definition to
ecs_service_fargate_elb
module - Added additional permission to Fargate task role policy to allow Datadog agent to perform autodiscovery.
- When upgrading your application, please apply these changes using nulogy-deployer locally before doing a deploy because Buildkite role does not have permissions to change task role policy:
cd app_worker
terragrunt plan -var docker_image=<YOUR_CURRENT_DEPLOYED_GIT_HASH>
- Add option to enable
containerInsights
for theecs_cluster
module
- Removed
cloudfront_s3
module group,cloudfront
module, andecs_service
module- these were copied to the GO project repository (the only project that was using it) for easier maintainability
- Updated the
ssl_policy
of thepublic_load_balancer
to use only TLSv1.2
- Remove ignore_changes from the ecs_fargate modules. This was used for autoscaling, though blocks non-autoscaling usage. Autoscaled support will need to be added properly in the future.
- Added an internal option for load balancers
- Extracted the terragrunt modules to their own repo
- Removes
cd_pipeline
module
- Fixes routes in vpc peering after upgrading to Terraform 0.12
- Fixes deprecation warning for
aws_lb_listener_rule.conditions
usage after upgrading aws provider to v2.59.0
- Migrates
aws_lb_listener_rule.conditions
inpublic_load_balancer
to new syntax since the old one was deprecated on v2.42.0 of the AWS provider
- Moves dockerignore file to docker build context root
- Fixes routes in vpc peering after upgrading to Terraform 0.12
- Bumps terragrunt
- Removes custom terraform-provider-aws because the bugfix from 8.19.0 got merged into the mainline
- Bumps terragrunt
- Updates the AWS terraform provider to fix a bug with creating RDS read replicas in shared VPCs
- Moves the Dockerfile to the root dir for more flexibility
- The auth script now looks for the first occurrence of 'nulogy-account-name' in your path rather than the last.
- ecs_incoming_allowed_cidr variable to ecs_fargate_with_elb. Useful for VPC Tunnels.
- Bumps Terraform and Terragrunt versions. Uses AWS Terraform provider 2.55.0 which fixes bug with restoring snapshots in shared VPCs.
- Added the
lb_listener_arns
output to theecs-fargate-with-elb
module
- Bumps Terraform (0.12.21) and Terragrunt (0.22.4) versions
- Added the
lb_cert_arn
variable to the "ecs_service_fargate_elb" module
- Added the
health_check_command
variable to the "ecs_service_fargate_elb" module
- empty_s3_bucket.sh script to help delete versioned s3 buckets
- Bumps Terraform and Terragrunt versions
- Bugfix for tgprep
- Terragrunt apply-all, plan-all and destroy-all aliases now work with symlink style modules.
- setup_auth.sh now shows your values as you paste them.
- auth.sh now caches account names for performance.
- auth.sh now uses the nulogy-anchor account to discover account names.
- auth.sh now automatically checks if you are in an aws account directory and uses that if the command line parameter hasn't been set.
- auth.sh and setup_auth.sh scripts which make it easy for users to login to the nulogy-auth account.
ecs_fargate_with_elb
now outputsecs_service_name
/deployer/utils/empty.hcl
placeholder for Terragrunt when you with to optionally include a file./deployer/utils/replace_terragrunt_hcl_with_symlinks.sh
Replaces regular environment hcl files with symlinks.
- Added optional
security_group_ids
topublic_load_balancer
module
- Office IP var to the ecs_plb_platform module group
- Added aliases for Terragrunt
tg=terragrunt
,tga=terragrunt apply
,tgpa=terragrunt plan-all
, etc...
- Bump Terraform and Terragrunt versions
- Updates default Nulogy office IP address
- Fixes security group for Public Load Balancer ipv6 support
- Public load balancer now supports ip_address_type "dualstack" for ipv6 support
- Bump Terraform and Terragrunt versions
- Added variable to set mutability behaviour on ECRs
- Added experimental lambda based autoscaling variable to buildkite stack
- Fixed lifecycle syntax for buildkite stack after upgrading to terraform 0.12
- Bump Terraform and Terragrunt versions
- Added health_check_timeout variable to the ecs_fargate_with_elb and to the public_load_balancer modules
- Added deregistration_delay variable to the ecs_fargate_with_elb and to the public_load_balancer modules to speed up deployments on some environments
- Removed HTTP 301 (redirect) as a healthy response for the load balancer target group.
- Added local provider to Dockerfile
- Fixes bug in modules/public_private_subnets/outputs.tf and modules/private_subnets/outputs.tf
- Bug fix for modules/ecs_service_fargate. Depends_on is a reserved name in terraform 0.12.
- Upgrades to Terraform 0.12.7 and Terragrunt 0.19.21.
- Updates modules to Terraform 0.12 format
To migrate a project's modules,
run: find ./* -type d -maxdepth 0 -exec terraform 0.12upgrade -yes {} \;
Convert terragrunt terraform.tfvars
files in the environment root and module
directories to terragrunt.hcl
- Removes Terraform Landscape as it's no longer needed with Terraform 0.12
- Added variable for
command
for ecs_service module container task definitions
- Uses a template file to configure ecs_service module container task definitions (no API change)
- Added parameter
scale_down_period
to the "buildkite_elastic_stack" module
- Added additional terraform providers.
- Added support for an extra security ingress to ECS Cluster. Defaults to 127.0.0.0/8 CIDR, port 65535 and UDP protocol.
- Added NAT Gateway public IPs as an output
Added Buildkite Agent Timestamp Lines (BuildkiteAgentTimestampLines)
Removed stack_ami_version
variable for buildkite_elastic_stack module since
it's not used
- Added missing deployer permissions
- Outputs ecs private subnets for fargate event shovel
- Uses skip variable for ecr lifecycle policy
- Removes code for rolling upgrades of ECS AMIs
- Changes cache_behaviour to ordered_cache_behaviour as cache_behaviour was removed in AWS provider version 2
- Public load balancer target group considers HTTP 301 (redirect) as a healthy
response. This allows
config.force_ssl
to be set as true in Rails.
- Updates terraform, terragrunt and landscape. Avoids re-downloading the aws provider.
- Sets the TERRAGRUNT_DOWNLOAD environment variable to /root/.terragrunt. This prevents the host's filesystem from getting cluttered with .terragrunt-cache directories.
- Fixes the priority values in order to avoid collisions on LBs routing rules during the terraform deployment.
- Fixes the default status code (503) for maintenance pages managed by AWS Load Balancer.
- Added support to maintenance pages on AWS Load Balancer level by means of ' aws_lb_listener_rule'. It can serve static content (Plain Text/HTML) content up to 1024 bytes.
- Added ssm:GetParameters permission to every role that was using ssm: GetParameter. Shockingly, these are two permissions different.
- Allow configuring alerting threshold for event_shovel.
- Allow passing in stickiness header to Fargate with Load Balancer.
- The nulogy deployer version is shown in the shell prompt.
- ECS fargate with elb module no longer hardcodes the security group name to "
PackManager App Workers".
- Existing users of this module can
set
security_group_name = "PackManager App Workers"
to avoid downtime changes.
- Existing users of this module can
set
-
Moved Event Shovel from EC2 servers to Fargate.
-
Required variables: "private_subnet_ids", "vpc_id" for Fargate support.
- ECR now has better default rules.
- ECR No longer relies on prefix tag to apply lifecycle policy.
- ECR variable
count_cap_tag_prefix
is now deprecated.
- Added slow_start option.
- Added ssm:PutParameter permission to Fargate clusters
- Added log_group_arn output to Fargate.
- Added log_group output to Fargate with ECS and a load balancer Module to allow shipping logs from Cloudwatch.
- Added Module for Fargate with ECS and a load balancer.
- Added missing permissions to the CI Pipeline Policy
Document:
ecr:ListTagsForResource
- Bug fix
- Changes ecs_service_fargate to use a json file, allowing multiple configurations
- Upgrade terragrunt from 0.14.7 to 0.17.1
- Module group "elastic_ci_stack" has been deleted as it made too many assumptions and is no longer used
- Module "buildkite_elastic_stack" has been modified to take a vpc and subnet to allow more flexibility
- Allow adding more to ECS service IAM role via output.
- Elastic_ci_stack builders can now specify a spot price.
- Added command variable to Fargate ECS module in order to allow the same container to be used for multiple purposes
- Added health_check variable to Fargate ECS module
- Uses cluster name instead of environment name to avoid name collision between ECS clusters on the same environment
- Fix bug with passing in Security Groups to Fargate module.
- Fargate support.
- Support for Fargate ECS Service with no load balancer (e.g. Background workers).
- ECR module outputs name now.
- Must pass in full string for AMI lookup now.
- For example, "2018.03.g" -> "amzn-ami-2018.03.g-amazon-ecs-optimized"
- Allow different owner for AMI lookup. Defaults to Amazon.
- Fix conflict between autoscaling cluster and rollingupdate cluster.
- Added a variable to customize root volume size of buildkite builders
- Moved hardcoded evaluation_periods to be a variable on event_shovel module group and event_shovel_ecs_service module
- Add optional Buildkite server size parameter to allow cutting the size for cost savings.
- Add optional stickiness header for CSS/JS.
- Fixed an issue where lambdas and iam roles for ECS clusters were colliding since they weren't namespaced
- Added the event_shovel module group
- ECS modules & module groups no longer support the
skip
variable. It was causing issues with terraform interpolation and wasn't actually used anywhere.
- Changed how ecs_auto_scaling_group works, to handle AMI / Launch Configuration changes gracefully without container downtime.
- Added ECS environment variables to support ECS metadata access from within the container
- Added builder_min_size, builder_scale_up_adjustment and builder_scale_down_adjustment optional variables to elastic_ci_stack module group
- Added scaled_down_adjustment and scale_up_adjustment to buildkite_elastic_stack module
- Add module for building and deploying a RabbitMQ cluster
- Additional configuration for elastic ci stack builders
- Output ECS autoscaling group name to make autoscaling easier.
- Allow overriding default_cooldown for autoscaling rules.
- Ignore changes on the desired_capacity, so it doesn't revert after autoscaling.
- Allow using buildkite_elastic_stack with a bootstrap script policy.
- Change the pathing for modules to allow apps to use subfolders in their directory structure.
- Switch the placement strategy for ECS containers to spread by availability zone.
- Make the creation of
aws_ecr_lifecycle_policy
optional
- Decouple
buildkite_queue
frombuildkite_env_name
so we can support blue/green deployments of buildkite stack
- Add an optional profile parameter to the cloudfront module. Backwards compatible.
- Add a builder_bootstrap_script_url to the elastic_ci_stack module
- Add support for v3.2 of the buildkite AWS stack
Moves the docker repository from ECR to our publicly hosted Docker Hub repo.
- Add a security group to buildkite stack so only office ip can SSH to it
- Add output for runners_vpc_id, runners_public_subnet_ids and runners_security_group_id for the Buildkite stack
- Bootstrap script url for buildkite runners
- Add runners_agents_per_instance variable #49
- Move creation of secret bucket outside of nulogy deployer #49
- Remove
tg_aws.sh
utility script #49
- Clean up postgres config (Evan Brodie) #47
- Added tg_aws.sh utility to make it easier to run aws commands with the right profile and region
- Added ECR Lifecycle policy to ECR module that keeps newest 100 images only
- tg_deploy.sh, tg_prepare.sh, tg_setup.sh and tg_teardown.sh are updated to properly handle resources and not just modules
- buildkite elastic ci stack: Move ECR build repo out of the module group
- Make stack_ami_version an optional variable in elastic_ci_stack module group
- A
stack_ami_version
variable for the buildkite stack
- Module for buildkite elastic stack
- Upgrade terraform from 0.11.1 to 0.11.7
- Upgrade terragrunt from 0.13.23 to 0.14.7
- Move Terraform code in
apps
to their app repos