Terraform apply does not delete Route53 record set while it says 'Resources: 1 added, 0 changed, 1 destroyed.' #9024

MBali-GitHub · 2019-06-17T19:05:14Z

I have updated a Route53 record Set which needed the resource replacement. I get a success message from terraform apply for the same. But looking at Route 53 in AWS, i can see that the new record set was created but the old one was not deleted.

# aws_route53_record.recordSet must be replaced

$ terraform apply -input=false "planfile"
aws_route53_record.recordSet: Destroying... [id=xxxxxxxxxxx]
aws_route53_record.recordSet: Destruction complete after 1s
aws_route53_record.recordSet: Creating...
aws_route53_record.recordSet: Still creating... [10s elapsed]
aws_route53_record.recordSet: Still creating... [20s elapsed]
aws_route53_record.recordSet: Still creating... [30s elapsed]
aws_route53_record.recordSet: Creation complete after 38s [id=xxxxxxxxxxx]

Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

KimSamba · 2019-07-01T09:16:04Z

I have exactly the same issue, I need to delete the records manually after, otherwise i can have issues where i try to recreate it, but it already exists...

grahamhar · 2019-07-18T07:04:56Z

Adding some detail from my observation of this. I changed the domain that the records resided in, not sure if that has any relevance.

terraform version
Terraform v0.12.3

provider.aws v2.16.0

aws_route53_record.wiremock[0]: Destroying... [id=XXXXXXXXX_www.redacted.com_CNAME]
aws_route53_record.wiremock[0]: Destruction complete after 2s
aws_route53_record.wiremock[0]: Creating...
aws_route53_record.wiremock[0]: Still creating... [20s elapsed]
aws_route53_record.wiremock[0]: Creation complete after 41s [id=YYYYYYYY_www.redacted.org_CNAME]

Mochizuki-S2 · 2019-07-18T12:03:44Z

Adding my observation too.

Terraform Version

terraform v0.12.0, v0.12.3, v0.12.4 (not happen on v0.11.14 )
provider.aws v2.19.0

Affected Resource(s)

aws_route53_record

Steps to Reproduce

create terraform.tf and terraform apply

$ cat terraform.tf
terraform {
}

provider "aws" {
  region  = "ap-northeast-1"
  version = "~> 2.0"
  profile = "sandbox"
}

resource "aws_route53_zone" "example" {
  name  = "my.foobar.example.com"
}

resource "aws_route53_record" "example" {
  zone_id = "${aws_route53_zone.example.id}"
  name    = "foo"
  records = ["1.1.1.1"]
  type    = "A"
  ttl     = 300
}

$ /tmp/terraform-v0.12.0 plan

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_route53_record.example will be created
  + resource "aws_route53_record" "example" {
      + allow_overwrite = (known after apply)
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "foo"
      + records         = [
          + "1.1.1.1",
        ]
      + ttl             = 300
      + type            = "A"
      + zone_id         = (known after apply)
    }
  # aws_route53_zone.example will be created
  + resource "aws_route53_zone" "example" {
      + comment       = "Managed by Terraform"
      + force_destroy = false
      + id            = (known after apply)
      + name          = "my.foobar.example.com"
      + name_servers  = (known after apply)
      + vpc_id        = (known after apply)
      + vpc_region    = (known after apply)
      + zone_id       = (known after apply)
    }

Plan: 2 to add, 0 to change, 0 to destroy.

$ /tmp/terraform-v0.12.0 apply -auto-approve

$ aws route53 list-resource-record-sets --hosted-zone-id Z1DIV15ETMOU37 --profile sandbox | jq '.ResourceRecordSets[].Name'
"my.foobar.example.com."
"my.foobar.example.com."
"foo.my.foobar.example.com."

change record name from foo to bar and terraform apply again

$ cat terraform.tf
terraform {
}

provider "aws" {
  region  = "ap-northeast-1"
  version = "~> 2.0"
  profile = "sandbox"
}

resource "aws_route53_zone" "example" {
  name  = "my.foobar.example.com"
}

resource "aws_route53_record" "example" {
  zone_id = "${aws_route53_zone.example.id}"
  # name    = "foo"
  name    = "bar"
  records = ["1.1.1.1"]
  type    = "A"
  ttl     = 300
}

$ /tmp/terraform-v0.12.0 plan

  # aws_route53_record.example must be replaced
-/+ resource "aws_route53_record" "example" {
      + allow_overwrite = (known after apply)
      ~ fqdn            = "foo.my.foobar.example.com" -> (known after apply)
      ~ id              = "Z1DIV15ETMOU37_foo_A" -> (known after apply)
      ~ name            = "foo" -> "bar" # forces replacement
        records         = [
            "1.1.1.1",
        ]
        ttl             = 300
        type            = "A"
        zone_id         = "Z1DIV15ETMOU37"
    }

Plan: 1 to add, 0 to change, 1 to destroy.

$ /tmp/terraform-v0.12.0 apply -auto-approve

list-resource-record-sets ( first record remains )

$ aws route53 list-resource-record-sets --hosted-zone-id Z1DIV15ETMOU37 --profile sandbox | jq '.ResourceRecordSets[].Name'
"my.foobar.example.com."
"my.foobar.example.com."
"bar.my.foobar.example.com."
"foo.my.foobar.example.com."

Expected Behavior

delete first route53 record (foo.my.foobar.example.com.)

Debug Output

before Destruction, findRecord not correct one (bar.my.foobar.example.com)

detail is around 2nd findRecord

$ TF_LOG=DEBUG /tmp/terraform-v0.12.0 apply -auto-approve
...
2019-07-18T19:34:20.277+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: 2019/07/18 19:34:20 [DEBUG] Expanded record name: foo.my.foobar.example.com
...
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: 2019/07/18 19:34:33 [DEBUG] Expanded record name: bar.my.foobar.example.com
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: 2019/07/18 19:34:33 [DEBUG] List resource records sets for zone: Z1DIV15ETMOU37, opts: {
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4:   HostedZoneId: "Z1DIV15ETMOU37",
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4:   MaxItems: "1",
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4:   StartRecordName: "bar.my.foobar.example.com.",
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4:   StartRecordType: "A"
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: }
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: 2019/07/18 19:34:33 [DEBUG] [aws-sdk-go] DEBUG: Request route53/ListResourceRecordSets Details:
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: GET /2013-04-01/hostedzone/Z1DIV15ETMOU37/rrset?maxitems=1&name=bar.my.foobar.example.com.&type=A HTTP/1.1
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: Host: route53.amazonaws.com
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: User-Agent: aws-sdk-go/1.20.17 (go1.12.6; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.2
...
aws_route53_record.example: Destruction complete after 2s
...
2019-07-18T19:35:13.413+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: 2019/07/18 19:35:13 [DEBUG] Expanded record name: bar.my.foobar.example.com
...

terraform v0.11.14 looks correct

around 2nd findRecord too

$ TF_LOG=DEBUG /tmp/terraform-v0.11.14 apply -auto-approve
...
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: 2019/07/18 20:49:06 [DEBUG] Expanded record name: foo.my.foobar.example.com
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: 2019/07/18 20:49:06 [DEBUG] List resource records sets for zone: Z392DQN8J7NMJQ, opts: {
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4:   HostedZoneId: "Z392DQN8J7NMJQ",
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4:   MaxItems: "1",
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4:   StartRecordName: "foo.my.foobar.example.com.",
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4:   StartRecordType: "A"
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: }
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: 2019/07/18 20:49:06 [DEBUG] [aws-sdk-go] DEBUG: Request route53/ListResourceRecordSets Details:
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: GET /2013-04-01/hostedzone/Z392DQN8J7NMJQ/rrset?maxitems=1&name=foo.my.foobar.example.com.&type=A HTTP/1.1
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: Host: route53.amazonaws.com
...
aws_route53_record.example: Destruction complete after 39s

axw-pivorra · 2019-08-09T09:33:08Z

Hi,
Same issue detected for CNAME and PTR route53 records.
Old records are never destroyed but terraform apply said that resources are destroyed.

vsalama · 2019-08-09T09:41:14Z

Hi,
same issue with:
terraform v0.12.6
provider.aws v2.23.0
but not with:
terraform v0.11.10
provider.aws v2.23.0

vsalama · 2019-08-09T13:40:56Z

and work also on:
terraform v0.11.14
provider.aws v2.23.0
end does not work on:
terraform v0.12.1
provider.aws v2.23.0
I think it's regression since terraform 12

stephendavidmarsh · 2019-08-12T21:27:55Z

I am also experiencing this issue, where old records are not deleted. In my case it also seems to not check if a record already exists under that name before updating the zone. As a result, it will write over records, including ones that were not managed by Terraform or leftover records from the deletion issue.

corymatheson · 2019-08-14T21:07:12Z

I am experiencing this issue as well. I was able to work around the issue by using the lifecycle hook in my records. lifecycle { create_before_destroy = true }

stephendavidmarsh · 2019-08-17T12:33:09Z

I did more testing around the writing over existing records behavior that I had observed. It doesn't actually update existing records, it deletes them, then creates a new record on top. This is obviously quite dangerous, because if you accidentally tell Terraform to create a new record with the same name as an old one, you will temporarily not have a record for that domain name.

With the workaround @NurSpass suggested (lifecycle { create_before_destroy = true }) it doesn't have this behavior. If it encounters an already existing record, it will stop and fail. When there is no existing record, it correctly deletes the old one like it should.

My theory is that Terraform, without create_before_destroy, is deleting the new record instead of the old one. This is both why it will destroy existing records and why it doesn't clean up the old one.

Mochizuki-S2 · 2019-09-03T13:03:25Z

I created pull request #9965

We might be able to avoid this issue to use not d.Get("name") but d.GetChange("name")

I confirmed the following.

I try printf debugging on findRecord as below when aws_route53_record.name forces replacement

i := 0
for {
        pt, file, line, ok := runtime.Caller(i)
        if !ok {
                break
        }
        funcName := runtime.FuncForPC(pt).Name()
        log.Printf("file=%s, line=%d, func=%v\n", file, line, funcName)
        i += 1
}

before_name, after_name := d.GetChange("name")
log.Printf("id=%s zone_id=%s name=%s change_name=%s,%s fqdn=%s type=%s\n", d.Id(), d.Get("zone_id"), d.Get("name"), before_name, after_name, d.Get("fqdn"), d.Get("type"))

then print as below

file=.../terraform-provider-aws/aws/resource_aws_route53_record.go, line=629, func=github.com/terraform-providers/terraform-provider-aws/aws.findRecord
file=.../github/terraform-provider-aws/aws/resource_aws_route53_record.go, line=526, func=github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsRoute53RecordRead
...
id=Z17UM5KN8LG9IH_foo_A zone_id=Z17UM5KN8LG9IH name=foo change_name=,foo fqdn= type=A
...
file=.../terraform-provider-aws/aws/resource_aws_route53_record.go, line=629, func=github.com/terraform-providers/terraform-provider-aws/aws.findRecord
file=.../terraform-provider-aws/aws/resource_aws_route53_record.go, line=737, func=github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsRoute53RecordDelete
...
id=Z17UM5KN8LG9IH_foo_A zone_id=Z17UM5KN8LG9IH name=bar change_name=foo,bar fqdn=foo.my.foobar.example.com type=A
...
file=.../terraform-provider-aws/aws/resource_aws_route53_record.go, line=629, func=github.com/terraform-providers/terraform-provider-aws/aws.findRecord
file=.../terraform-provider-aws/aws/resource_aws_route53_record.go, line=459, func=github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsRoute53RecordCreate
...
id=Z1LGSSII0WM0DD_bar_A zone_id=Z1LGSSII0WM0DD name=bar change_name=,bar fqdn= type=A

this time, I try chaning route53 zone too as below

resource "aws_route53_record" "example" {
  # zone_id = "${aws_route53_zone.example.id}"
  zone_id = "${aws_route53_zone.example2.id}"
  # name    = "foo"
  name    = "bar"
  records = ["1.1.1.1"]
  type    = "A"
  ttl     = 300
}

I think it may not be right that d.Get("name") returns new value when destroying.

But I'm not sure, so create workaroud pull request.

pimperator · 2020-04-08T23:07:20Z

also affected:
Terraform v0.12.24

provider.archive v1.3.0
provider.aws v2.56.0
provider.template v2.1.2

minhluantran017 · 2020-05-27T02:33:46Z

The same issue with Route53 record deletion:

Terraform 0.12.25
provider.aws: version = "~> 2.63"
provider.template: version = "~> 2.1"

gawbul · 2020-06-05T16:46:14Z

Having the same issue with Route53 record deletion:

Terraform version: 0.12.11
provider.aws version: "= 2.62.0"
provider.template version: "~> 2.1.2"
provider.archive version: "~> 1.3.0"
provider.external version: "~> 1.2.0"

Would be great to see #11335 merged in as a possible fix please? 👍 @bflad

bflad · 2020-07-01T13:08:50Z

The fix for this has been merged and will release with version 2.69.0 of the Terraform AWS Provider, likely tomorrow. Thanks to @jbergknoff-rival for the implementation. 👍

ghost · 2020-07-03T01:02:20Z

This has been released in version 2.69.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

ghost · 2020-07-31T17:10:13Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

nywilken added the service/route53 Issues and PRs that pertain to the route53 service. label Jun 17, 2019

aeschright added the needs-triage Waiting for first response or review from a maintainer. label Jun 24, 2019

bflad added bug Addresses a defect in current functionality. and removed needs-triage Waiting for first response or review from a maintainer. labels Jul 18, 2019

Mochizuki-S2 mentioned this issue Sep 3, 2019

Workaround for get r53record host name #9965

Closed

dav3r mentioned this issue Sep 11, 2019

Terraform bug impacting deployments cisagov/cyhy_amis#235

Open

bflad mentioned this issue Dec 5, 2019

Changing name of "A" record is incorrect #11156

Closed

Mochizuki-S2 mentioned this issue Dec 9, 2019

aws_route53_record.dns_record must be replaced but isn't #9688

Closed

jbergknoff-rival mentioned this issue Dec 17, 2019

Delete old Route53 record when updating name #11335

Merged

bflad added this to the v2.69.0 milestone Jul 1, 2020

This was referenced Jul 1, 2020

Terraform won't delete the hosted_zone and trying to recreate the DNS entry with every single apply #9485

Closed

Route53 PTR Records not deleted on taint #13234

Closed

bflad closed this as completed in #11335 Jul 1, 2020

ghost locked and limited conversation to collaborators Jul 31, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Terraform apply does not delete Route53 record set while it says 'Resources: 1 added, 0 changed, 1 destroyed.' #9024

Terraform apply does not delete Route53 record set while it says 'Resources: 1 added, 0 changed, 1 destroyed.' #9024

MBali-GitHub commented Jun 17, 2019

KimSamba commented Jul 1, 2019

grahamhar commented Jul 18, 2019

Mochizuki-S2 commented Jul 18, 2019

axw-pivorra commented Aug 9, 2019

vsalama commented Aug 9, 2019 •

edited

Loading

vsalama commented Aug 9, 2019 •

edited

Loading

stephendavidmarsh commented Aug 12, 2019

corymatheson commented Aug 14, 2019

stephendavidmarsh commented Aug 17, 2019

Mochizuki-S2 commented Sep 3, 2019 •

edited

Loading

pimperator commented Apr 8, 2020

minhluantran017 commented May 27, 2020

gawbul commented Jun 5, 2020

bflad commented Jul 1, 2020

ghost commented Jul 3, 2020

ghost commented Jul 31, 2020

Terraform apply does not delete Route53 record set while it says 'Resources: 1 added, 0 changed, 1 destroyed.' #9024

Terraform apply does not delete Route53 record set while it says 'Resources: 1 added, 0 changed, 1 destroyed.' #9024

Comments

MBali-GitHub commented Jun 17, 2019

KimSamba commented Jul 1, 2019

grahamhar commented Jul 18, 2019

Mochizuki-S2 commented Jul 18, 2019

Terraform Version

Affected Resource(s)

Steps to Reproduce

Expected Behavior

Debug Output

axw-pivorra commented Aug 9, 2019

vsalama commented Aug 9, 2019 • edited Loading

vsalama commented Aug 9, 2019 • edited Loading

stephendavidmarsh commented Aug 12, 2019

corymatheson commented Aug 14, 2019

stephendavidmarsh commented Aug 17, 2019

Mochizuki-S2 commented Sep 3, 2019 • edited Loading

pimperator commented Apr 8, 2020

minhluantran017 commented May 27, 2020

gawbul commented Jun 5, 2020

bflad commented Jul 1, 2020

ghost commented Jul 3, 2020

ghost commented Jul 31, 2020

vsalama commented Aug 9, 2019 •

edited

Loading

vsalama commented Aug 9, 2019 •

edited

Loading

Mochizuki-S2 commented Sep 3, 2019 •

edited

Loading