r/s3_bucket_lifecycle_configuration: providing filter "and" clause with only "prefix" causes MalformedXML error

[Nuru]

With the combination of this issue, #23883, and #23884 I have no way to fully use and maintain S3 lifecycle configuration rules.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.1.6
on darwin_amd64

• provider registry.terraform.io/hashicorp/aws v4.8.0

Affected Resource(s)

• s3_bucket_lifecycle_configuration

Terraform Configuration Files

provider "aws" {
  region = "us-east-2"
}

resource "aws_s3_bucket" "example" {
  bucket = "aws-provider-issue-23882"
}

resource "aws_s3_bucket_versioning" "example" {
  bucket = aws_s3_bucket.example.id

  versioning_configuration {
    status = "Enabled"
  }
}

# https://github.com/hashicorp/terraform-provider-aws/issues/23375
resource "aws_s3_bucket_lifecycle_configuration" "example" {
  bucket = aws_s3_bucket.example.id

  rule {
    id     = "rule"
    status = "Enabled"

    filter {
      and {
        prefix = "prefix1"
      }
    }

    expiration {
      days = 90
    }
}

output "lifecycle" {
  value = jsondecode(jsonencode(aws_s3_bucket_lifecycle_configuration.example.rule))
}

Debug Output

Here is the relevant PUT content and response (most headers removed):

2022-03-25T18:06:15.479-0700 [DEBUG] provider.terraform-provider-aws_v4.8.0_x5: [aws-sdk-go] DEBUG: Request s3/PutBucketLifecycleConfiguration Details:
---[ REQUEST POST-SIGN ]-----------------------------
PUT /?lifecycle= HTTP/1.1
Host: aws-provider-issue-23882.s3.us-east-2.amazonaws.com
Accept-Encoding: gzip

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Rule><Status>Enabled</Status><Expiration><Days>90</Days></Expiration><Filter><And><Prefix>prefix1</Prefix></And></Filter><ID>rule</ID></Rule></LifecycleConfiguration>
-----------------------------------------------------: timestamp=2022-03-25T18:06:15.479-0700
2022-03-25T18:06:15.936-0700 [DEBUG] provider.terraform-provider-aws_v4.8.0_x5: [aws-sdk-go] DEBUG: Response s3/PutBucketLifecycleConfiguration Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 400 Bad Request

Expected Behavior

S3 bucket with lifecycle configuration should be created without error.

Actual Behavior

│ Error: error creating S3 Lifecycle Configuration for bucket (aws-provider-issue-23882): MalformedXML: The XML you provided was not well-formed or did not validate against our published schema

Steps to Reproduce

1. terraform init -upgrade
2. terraform apply -auto-approve

Important Factoids

The filter takes an optional and configuration block which takes any of 4 optional arguments:

• object_size_greater_than
• object_size_less_than
• prefix
• tags

It appears to work if you supply only one of any of the arguments except for prefix. If you only specify prefix, you get the MalformedXML error. This is inconsistent, unnecessarily complex, and adds further confusion after the deprecation of prefix at other levels of the resources. Based on the comments on #23299 I though it would be safest to always put the prefix in the and block but even that doesn't work.

If I try to work around this issue by changing from filter { and { to filter { prefx = then I trigger #23884 and #23883. So this is a blocker.

References

• Bad error (message?) aws_s3_bucket_lifecycle_configuration complains about "Base level prefix" when there is none #23299
• r/s3_bucket_lifecycle_configuration: improve documentation and officially deprecate prefix #23325
• r/s3_bucket_lifecycle_configuration: Provider produced inconsistent final plan #23883
• r/s3_bucket_lifecycle_configuration: Rule change generates MalformedXML #23884

[anGie44]

Hi @Nuru , thank you for raising this issue and apologies you've come across some confusing behavior. One thing to note is that to use the filter.and configuration block, you must include at least 2 predicates per the AWS API docs (https://docs.aws.amazon.com/AmazonS3/latest/API/API_LifecycleRuleAndOperator.html) which explains the MalformedXML error, though I can see we're not consistent when using filter.and.prefix vs. the other optional params like filter.and.object_size_greater_than.

It appears to work if you supply only one of any of the arguments except for prefix. If you only specify prefix, you get the MalformedXML error.

This occurs because we include a filter.and.prefix value in the PUT request even if it's an empty string so this could be a bug or we should at least make this apparent in the resource documentation. The portion of the code that is responsible for this behavior can be seen here:

terraform-provider-aws/internal/service/s3/flex.go

Lines 262 to 289 in a0cf29f

	func ExpandLifecycleRuleFilterAndOperator(m map[string]interface{}) *s3.LifecycleRuleAndOperator {
	if len(m) == 0 {
	return nil
	}
	result := &s3.LifecycleRuleAndOperator{}
	if v, ok := m["object_size_greater_than"].(int); ok && v > 0 {
	result.ObjectSizeGreaterThan = aws.Int64(int64(v))
	}
	if v, ok := m["object_size_less_than"].(int); ok && v > 0 {
	result.ObjectSizeLessThan = aws.Int64(int64(v))
	}
	if v, ok := m["prefix"].(string); ok {
	result.Prefix = aws.String(v)
	}
	if v, ok := m["tags"].(map[string]interface{}); ok && len(v) > 0 {
	tags := Tags(tftags.New(v).IgnoreAWS())
	if len(tags) > 0 {
	result.Tags = tags
	}
	}
	return result
	}

where the relevant portion is:

if v, ok := m["prefix"].(string); ok { 
 		result.Prefix = aws.String(v) 
 	} 

I'll have a look at #23883 and #23884 as I would of expected filter { prefix = "prefix1" } to work as-is.

[Nuru]

Given #23884, I suggest a workaround of always including ObjectSizeGreaterThan 0 or empty Tags or something like that so that the and block always has at least 2 predicates.

[dignajar]

I have the same issue, with AWS provider 4.38.0.

  # aws_s3_bucket_lifecycle_configuration.s3_backups will be updated in-place
  ~ resource "aws_s3_bucket_lifecycle_configuration" "s3_backups" {
        id     = "backups"
        # (1 unchanged attribute hidden)

      ~ rule {
            id     = "retention14d"
            # (1 unchanged attribute hidden)

          ~ filter {
                # (1 unchanged attribute hidden)

              + and {
                  + object_size_greater_than = 0
                }
            }

            # (1 unchanged block hidden)
        }

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

aws_s3_bucket_lifecycle_configuration.s3_backups: Modifying... [id=backups]
╷
│ Error: error updating S3 Bucket Lifecycle Configuration (backups): MalformedXML: The XML you provided was not well-formed or did not validate against our published schema
│       status code: 400, request id: [redacted], host id: [redacted]
│ 
│   with aws_s3_bucket_lifecycle_configuration.s3_backups,
│   on main.tf line 61, in resource "aws_s3_bucket_lifecycle_configuration" "s3_backups":
│   61: resource "aws_s3_bucket_lifecycle_configuration" "s3_backups" {
│ 
╵

The Terraform code

  rule {
    ...
    filter {
      prefix = "test/"
      and {
        object_size_greater_than = 0
      }
    }
    ...
  }

Also I tried

  rule {
    ...
    filter {
      and {
        prefix = "test/"
        object_size_greater_than = 0
      }
    }
    ...
  }

Same issue with the MalformedXML error.

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[Nuru]

Can we please get this fixed already?

[daltschu22]

This issue is affecting us aswell.