Bad error (message?) aws_s3_bucket_lifecycle_configuration complains about "Base level prefix" when there is none

[Nuru]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.1.6
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v4.2.0
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/time v0.7.2

Affected Resource(s)

• aws_s3_bucket_lifecycle_configuration

Terraform Configuration Files

Creating an S3 bucket and aws_s3_bucket_lifecycle_configuration but only specifying prefix inside the rule.filter block.

Click to reveal `aws_s3_bucket_lifecycle_configuration` code

provider "aws" {
  region = "us-east-2"
}

locals {
  example_rules = [{
    id      = "rule-1"
    prefix  = null
    enabled = true
    tags    = { "temp" : "true" }

    enable_noncurrent_version_expiration = true

    abort_incomplete_multipart_upload_days = null
    noncurrent_version_expiration_days     = 5
    },
    {
      id      = "rule-2"
      prefix  = null
      enabled = true
      tags    = {}

      enable_noncurrent_version_expiration = true

      abort_incomplete_multipart_upload_days = 2
      noncurrent_version_expiration_days     = 30
  }]
}

resource "aws_s3_bucket" "example" {
  bucket = "aws-provider-issue-23299"
}

resource "aws_s3_bucket_lifecycle_configuration" "example" {
  count  = 1
  bucket = aws_s3_bucket.example.id

  dynamic "rule" {
    for_each = local.example_rules

    content {
      id     = rule.value.id
      status = try(rule.value.enabled == true ? "Enabled" : "Disabled", rule.value.status)

      dynamic "filter" {
        for_each = (try(length(rule.value.prefix), 0) + try(length(rule.value.tags), 0)) > 0 ? [1] : []
        content {
          prefix = rule.value.prefix
          dynamic "and" {
            for_each = try(length(rule.value.tags), 0) > 0 ? [1] : []
            content {
              tags = rule.value.tags
            }
          }
        }
      }
      dynamic "abort_incomplete_multipart_upload" {
        for_each = try(tonumber(rule.value.abort_incomplete_multipart_upload_days), null) != null ? [1] : []
        content {
          days_after_initiation = rule.value.abort_incomplete_multipart_upload_days
        }
      }


      dynamic "noncurrent_version_expiration" {
        for_each = rule.value.enable_noncurrent_version_expiration ? [1] : []

        content {
          noncurrent_days = rule.value.noncurrent_version_expiration_days
        }
      }
    }
  }
}

Expected Behavior

I expect this should apply cleanly. Note that if I apply only 1 of the 2 rules (either one) it does apply cleanly. It is only when applying both rules that it fails.

Actual Behavior

Error: error creating S3 Lifecycle Configuration for bucket (aws-provider-issue-23299): 
InvalidRequest: Base level prefix cannot be used in Lifecycle V2, prefixes are only supported in the Filter.

Steps to Reproduce

1. terraform apply

Important Factoids

BTW, it would be nice if the documentation would explain the difference between

rule {
  filter {
    prefix = "/foo"
    and {
      tags = { tagged = "yes" }
    }}}

and

rule {
  filter {
    and {
      prefix = "/foo"
      tags = { tagged = "yes" }
    }}}

References

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_lifecycle_configuration#filter

[anGie44]

Hi @Nuru , thank you for raising this issue. From other issues I've seen come in related to the aws_s3_bucket_lifecycle_configuration resource, I believe this stems from using filter in the first rule with id of rule-1 , while the second rule with id rule-2 does not have any filter specified thus this will default back to prefix as noted in the documentation. So to send a request that the upstream API will be happy with, both rules should use Lifecycle V2 syntax, where the second rule in this case would be configured with just filter {} if the rule is intended to be applied to all objects. Locally i've tweaked the provided configuration in the description to use filter {} for the second rule and terraform apply completes without error so that could be one approach, again assuming the rule is intended for all objects.

As for the documentation, I'll take note of the feedback as it can certainly be improved. Though i do want to note that

rule {
  filter {
    prefix = "/foo"
    and {
      tags = { tagged = "yes" }
    }}}

should technically not work as the S3 API documentation notes that

The Filter is used to identify objects that a Lifecycle Rule applies to. A Filter must have exactly one of Prefix, Tag, or And specified.

and within the resource, prefix is only specified in the API request if both tags and and are nil

terraform-provider-aws/internal/service/s3/flex.go

Line 245 in 19ee8ca

if v, ok := m["prefix"].(string); ok && result.And == nil && result.Tag == nil {

Thus an initial terraform apply will actually succeed but a subsequent terraform plan will show a diff like

 ~ filter {
              + prefix = "/foo"

                # (1 unchanged block hidden)
            }

[Nuru]

@anGie44 Thank you for the quick and detailed response.

It strikes me as odd (certainly unintuitive) that an empty filter block would be required, especially after #23228 was supposedly fixed. I have verified that supplying the empty filter works, but I would not close this issue because of that. I would convert it into a documentation request, giving specific guidance about this, since I'm not the only one who got tripped up by it.

[anGie44]

The resolution for #23228 is quite similar to here I believe in that the AWS API actually requires filter (Lifecycle V2 syntax) but users who were coming from the aws_s3_bucket resource (pre v4.0) were previously configuring rules with prefix (technically V1 syntax, but behind the scenes the API request was built with Lifecycle V2), and thus were not including filter in their translation from aws_s3_bucket to aws_s3_bucket_lifecycle_configuration.

I understand the existing guidance is not clear and will move forward with keeping this as a documentation update 👍 Thanks again @Nuru for the feedback. If you have recommendations for specific wording you would like to see, let us know or if you're interested in submitting a PR, happy to review it as well.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.