S3 Replication Improvements (#93)

Co-authored-by: Nuru <[email protected]>
Co-authored-by: Yonatan Koren <[email protected]>

README.md

@@ -28,15 +28,19 @@
 
 -->
 
-This module creates an S3 bucket with support of versioning, encryption, ACL and bucket object policy.
+This module creates an S3 bucket with support of versioning, replication, encryption, ACL, and bucket object policy.
 If `user_enabled` variable is set to `true`, the module will provision a basic IAM user with permissions to access the bucket.
 
-This basic IAM system user is suitable for CI/CD systems (_e.g._ TravisCI, CircleCI) or systems which are *external* to AWS that cannot leverage [AWS IAM Instance Profiles](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html).
+This basic IAM system user is suitable for CI/CD systems (_e.g._ TravisCI, CircleCI) or systems which are *external* to AWS that cannot leverage
+[AWS IAM Instance Profiles](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html) and
+do not already have IAM credentials. Users or systems that have IAM credentials should either be granted access directly based on
+their IAM identity or be allowed to assume an IAM role with access.
 
 We do not recommend creating IAM users this way for any other purpose.
 
-It blocks public access to the bucket by default.
-https://docs.aws.amazon.com/AmazonS3/latest/dev/access-control-block-public-access.html
+This module blocks public access to the bucket by default. See `block_public_acls`, `block_public_policy`,
+and `ignore_public_acls` to change the settings. See [AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/dev/access-control-block-public-access.html)
+for more details.
 
 ---
 
@@ -241,10 +245,12 @@ Available targets:
 | <a name="input_object_lock_configuration"></a> [object\_lock\_configuration](#input\_object\_lock\_configuration) | A configuration for S3 object locking. With S3 Object Lock, you can store objects using a `write once, read many` (WORM) model. Object Lock can help prevent objects from being deleted or overwritten for a fixed amount of time or indefinitely. | <pre>object({<br>    mode  = string # Valid values are GOVERNANCE and COMPLIANCE.<br>    days  = number<br>    years = number<br>  })</pre> | `null` | no |
 | <a name="input_policy"></a> [policy](#input\_policy) | A valid bucket policy JSON document. Note that if the policy document is not specific enough (but still valid), Terraform may view the policy as constantly changing in a terraform plan. In this case, please make sure you use the verbose/specific version of the policy | `string` | `""` | no |
 | <a name="input_regex_replace_chars"></a> [regex\_replace\_chars](#input\_regex\_replace\_chars) | Regex to replace chars with empty string in `namespace`, `environment`, `stage` and `name`.<br>If not set, `"/[^a-zA-Z0-9-]/"` is used to remove all characters other than hyphens, letters and digits. | `string` | `null` | no |
-| <a name="input_replication_rules"></a> [replication\_rules](#input\_replication\_rules) | Specifies the replication rules if S3 bucket replication is enabled | `list(any)` | `null` | no |
+| <a name="input_replication_rules"></a> [replication\_rules](#input\_replication\_rules) | DEPRECATED: Use s3\_replication\_rules instead. | `list(any)` | `null` | no |
 | <a name="input_restrict_public_buckets"></a> [restrict\_public\_buckets](#input\_restrict\_public\_buckets) | Set to `false` to disable the restricting of making the bucket public | `bool` | `true` | no |
-| <a name="input_s3_replica_bucket_arn"></a> [s3\_replica\_bucket\_arn](#input\_s3\_replica\_bucket\_arn) | The ARN of the S3 replica bucket (destination) | `string` | `""` | no |
-| <a name="input_s3_replication_enabled"></a> [s3\_replication\_enabled](#input\_s3\_replication\_enabled) | Set this to true and specify `s3_replica_bucket_arn` to enable replication. `versioning_enabled` must also be `true`. | `bool` | `false` | no |
+| <a name="input_s3_replica_bucket_arn"></a> [s3\_replica\_bucket\_arn](#input\_s3\_replica\_bucket\_arn) | A single S3 bucket ARN to use for all replication rules.<br>Note: The destination bucket can be specified in the replication rule itself<br>(which allows for multiple destinations), in which case it will take precedence over this variable. | `string` | `""` | no |
+| <a name="input_s3_replication_enabled"></a> [s3\_replication\_enabled](#input\_s3\_replication\_enabled) | Set this to true and specify `s3_replication_rules` to enable replication. `versioning_enabled` must also be `true`. | `bool` | `false` | no |
+| <a name="input_s3_replication_rules"></a> [s3\_replication\_rules](#input\_s3\_replication\_rules) | Specifies the replication rules for S3 bucket replication if enabled. You must also set s3\_replication\_enabled to true. | `list(any)` | `null` | no |
+| <a name="input_s3_replication_source_roles"></a> [s3\_replication\_source\_roles](#input\_s3\_replication\_source\_roles) | Cross-account IAM Role ARNs that will be allowed to perform S3 replication to this bucket (for replication within the same AWS account, it's not necessary to adjust the bucket policy). | `list(string)` | `[]` | no |
 | <a name="input_sse_algorithm"></a> [sse\_algorithm](#input\_sse\_algorithm) | The server-side encryption algorithm to use. Valid values are `AES256` and `aws:kms` | `string` | `"AES256"` | no |
 | <a name="input_stage"></a> [stage](#input\_stage) | Stage, e.g. 'prod', 'staging', 'dev', OR 'source', 'build', 'test', 'deploy', 'release' | `string` | `null` | no |
 | <a name="input_tags"></a> [tags](#input\_tags) | Additional tags (e.g. `map('BusinessUnit','XYZ')` | `map(string)` | `{}` | no |

README.yaml

@@ -66,15 +66,19 @@ related:
 
 # Short description of this project
 description: |-
-  This module creates an S3 bucket with support of versioning, encryption, ACL and bucket object policy.
+  This module creates an S3 bucket with support of versioning, replication, encryption, ACL, and bucket object policy.
   If `user_enabled` variable is set to `true`, the module will provision a basic IAM user with permissions to access the bucket.
 
-  This basic IAM system user is suitable for CI/CD systems (_e.g._ TravisCI, CircleCI) or systems which are *external* to AWS that cannot leverage [AWS IAM Instance Profiles](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html).
+  This basic IAM system user is suitable for CI/CD systems (_e.g._ TravisCI, CircleCI) or systems which are *external* to AWS that cannot leverage
+  [AWS IAM Instance Profiles](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html) and
+  do not already have IAM credentials. Users or systems that have IAM credentials should either be granted access directly based on
+  their IAM identity or be allowed to assume an IAM role with access.
 
   We do not recommend creating IAM users this way for any other purpose.
 
-  It blocks public access to the bucket by default.
-  https://docs.aws.amazon.com/AmazonS3/latest/dev/access-control-block-public-access.html
+  This module blocks public access to the bucket by default. See `block_public_acls`, `block_public_policy`,
+  and `ignore_public_acls` to change the settings. See [AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/dev/access-control-block-public-access.html)
+  for more details.
 
 # How to use this project
 usage: |-

docs/terraform.md

@@ -69,10 +69,12 @@
 | <a name="input_object_lock_configuration"></a> [object\_lock\_configuration](#input\_object\_lock\_configuration) | A configuration for S3 object locking. With S3 Object Lock, you can store objects using a `write once, read many` (WORM) model. Object Lock can help prevent objects from being deleted or overwritten for a fixed amount of time or indefinitely. | <pre>object({<br>    mode  = string # Valid values are GOVERNANCE and COMPLIANCE.<br>    days  = number<br>    years = number<br>  })</pre> | `null` | no |
 | <a name="input_policy"></a> [policy](#input\_policy) | A valid bucket policy JSON document. Note that if the policy document is not specific enough (but still valid), Terraform may view the policy as constantly changing in a terraform plan. In this case, please make sure you use the verbose/specific version of the policy | `string` | `""` | no |
 | <a name="input_regex_replace_chars"></a> [regex\_replace\_chars](#input\_regex\_replace\_chars) | Regex to replace chars with empty string in `namespace`, `environment`, `stage` and `name`.<br>If not set, `"/[^a-zA-Z0-9-]/"` is used to remove all characters other than hyphens, letters and digits. | `string` | `null` | no |
-| <a name="input_replication_rules"></a> [replication\_rules](#input\_replication\_rules) | Specifies the replication rules if S3 bucket replication is enabled | `list(any)` | `null` | no |
+| <a name="input_replication_rules"></a> [replication\_rules](#input\_replication\_rules) | DEPRECATED: Use s3\_replication\_rules instead. | `list(any)` | `null` | no |
 | <a name="input_restrict_public_buckets"></a> [restrict\_public\_buckets](#input\_restrict\_public\_buckets) | Set to `false` to disable the restricting of making the bucket public | `bool` | `true` | no |
-| <a name="input_s3_replica_bucket_arn"></a> [s3\_replica\_bucket\_arn](#input\_s3\_replica\_bucket\_arn) | The ARN of the S3 replica bucket (destination) | `string` | `""` | no |
-| <a name="input_s3_replication_enabled"></a> [s3\_replication\_enabled](#input\_s3\_replication\_enabled) | Set this to true and specify `s3_replica_bucket_arn` to enable replication. `versioning_enabled` must also be `true`. | `bool` | `false` | no |
+| <a name="input_s3_replica_bucket_arn"></a> [s3\_replica\_bucket\_arn](#input\_s3\_replica\_bucket\_arn) | A single S3 bucket ARN to use for all replication rules.<br>Note: The destination bucket can be specified in the replication rule itself<br>(which allows for multiple destinations), in which case it will take precedence over this variable. | `string` | `""` | no |
+| <a name="input_s3_replication_enabled"></a> [s3\_replication\_enabled](#input\_s3\_replication\_enabled) | Set this to true and specify `s3_replication_rules` to enable replication. `versioning_enabled` must also be `true`. | `bool` | `false` | no |
+| <a name="input_s3_replication_rules"></a> [s3\_replication\_rules](#input\_s3\_replication\_rules) | Specifies the replication rules for S3 bucket replication if enabled. You must also set s3\_replication\_enabled to true. | `list(any)` | `null` | no |
+| <a name="input_s3_replication_source_roles"></a> [s3\_replication\_source\_roles](#input\_s3\_replication\_source\_roles) | Cross-account IAM Role ARNs that will be allowed to perform S3 replication to this bucket (for replication within the same AWS account, it's not necessary to adjust the bucket policy). | `list(string)` | `[]` | no |
 | <a name="input_sse_algorithm"></a> [sse\_algorithm](#input\_sse\_algorithm) | The server-side encryption algorithm to use. Valid values are `AES256` and `aws:kms` | `string` | `"AES256"` | no |
 | <a name="input_stage"></a> [stage](#input\_stage) | Stage, e.g. 'prod', 'staging', 'dev', OR 'source', 'build', 'test', 'deploy', 'release' | `string` | `null` | no |
 | <a name="input_tags"></a> [tags](#input\_tags) | Additional tags (e.g. `map('BusinessUnit','XYZ')` | `map(string)` | `{}` | no |

examples/complete/fixtures.us-west-1.tfvars → examples/complete/fixtures.us-east-2.tfvars

@@ -1,6 +1,6 @@
 enabled = true
 
-region = "us-west-1"
+region = "us-east-2"
 
 namespace = "eg"
 

examples/complete/grants.us-west-1.tfvars → examples/complete/grants.us-east-2.tfvars

@@ -1,4 +1,4 @@
-region = "us-west-1"
+region = "us-east-2"
 
 namespace = "eg"
 

examples/complete/lifecycle.us-west-1.tfvars → examples/complete/lifecycle.us-east-2.tfvars

@@ -1,4 +1,4 @@
-region = "us-west-1"
+region = "us-east-2"
 
 namespace = "eg"
 

examples/complete/main.tf

@@ -1,3 +1,15 @@
+locals {
+  replication_enabled = length(var.s3_replication_rules) > 0
+  extra_rule = local.replication_enabled ? {
+    id                 = "replication-test-explicit-bucket"
+    status             = "Enabled"
+    prefix             = "/extra"
+    priority           = 5
+    destination_bucket = module.s3_bucket_replication_target_extra[0].bucket_arn
+  } : null
+  s3_replication_rules = local.replication_enabled ? concat(var.s3_replication_rules, [local.extra_rule]) : null
+}
+
 provider "aws" {
   region = var.region
 }
@@ -15,6 +27,39 @@ module "s3_bucket" {
   allowed_bucket_actions       = var.allowed_bucket_actions
   bucket_name                  = var.bucket_name
   object_lock_configuration    = var.object_lock_configuration
+  s3_replication_enabled       = local.replication_enabled
+  s3_replica_bucket_arn        = join("", module.s3_bucket_replication_target.*.bucket_arn)
+  s3_replication_rules         = local.s3_replication_rules
 
   context = module.this.context
 }
+
+module "s3_bucket_replication_target" {
+  count = local.replication_enabled ? 1 : 0
+
+  source = "../../"
+
+  user_enabled                = true
+  acl                         = "private"
+  force_destroy               = true
+  versioning_enabled          = true
+  s3_replication_source_roles = [module.s3_bucket.replication_role_arn]
+
+  attributes = ["target"]
+  context    = module.this.context
+}
+
+module "s3_bucket_replication_target_extra" {
+  count = local.replication_enabled ? 1 : 0
+
+  source = "../../"
+
+  user_enabled                = true
+  acl                         = "private"
+  force_destroy               = true
+  versioning_enabled          = true
+  s3_replication_source_roles = [module.s3_bucket.replication_role_arn]
+
+  attributes = ["target", "extra"]
+  context    = module.this.context
+}

examples/complete/object-lock.us-west-1.tfvars → examples/complete/object-lock.us-east-2.tfvars

@@ -1,4 +1,4 @@
-region = "us-west-1"
+region = "us-east-2"
 
 namespace = "eg"
 

examples/complete/outputs.tf

@@ -13,6 +13,21 @@ output "bucket_arn" {
   description = "Bucket ARN"
 }
 
+output "replication_bucket_id" {
+  value       = local.replication_enabled ? join("", module.s3_bucket_replication_target.*.bucket_id) : null
+  description = "Bucket Name (aka ID)"
+}
+
+output "replication_bucket_arn" {
+  value       = local.replication_enabled ? join("", module.s3_bucket_replication_target.*.bucket_arn) : null
+  description = "Bucket ARN"
+}
+
+output "replication_role_arn" {
+  value       = module.s3_bucket.replication_role_arn
+  description = "The ARN of the replication IAM Role"
+}
+
 output "bucket_region" {
   value       = module.s3_bucket.bucket_region
   description = "Bucket region"

examples/complete/replication.us-east-2.tfvars

@@ -0,0 +1,29 @@
+# Do not set "enabled", will be set by test framework
+# enabled                      = true
+region                       = "us-east-2"
+namespace                    = "eg"
+stage                        = "test"
+name                         = "s3-replication-test"
+acl                          = "private"
+force_destroy                = true
+versioning_enabled           = true
+allow_encrypted_uploads_only = true
+allowed_bucket_actions = [
+  "s3:PutObject",
+  "s3:PutObjectAcl",
+  "s3:GetObject",
+  "s3:DeleteObject",
+  "s3:ListBucket",
+  "s3:ListBucketMultipartUploads",
+  "s3:GetBucketLocation",
+  "s3:AbortMultipartUpload",
+]
+
+# Rules will be augmented with an additional bucket rule, so prefix cannot be "/"
+s3_replication_rules = [
+  {
+    id     = "replication-test"
+    status = "Enabled"
+    prefix = "/main"
+  }
+]

examples/complete/variables.tf

@@ -61,6 +61,11 @@ variable "lifecycle_rules" {
   description = "A list of lifecycle rules."
 }
 
+variable "s3_replication_rules" {
+  default     = []
+  description = "S3 replication rules"
+}
+
 variable "policy" {
   type        = string
   default     = ""