aws_security_group_rule not behaving idempotently

[jtopper]

If I add two SG rules to an SG, where the rules differ only by source security group id, Terraform gets confused on a second run and attempts to make changes where none are necessary.

Test case:

resource "aws_vpc" "test" {
    cidr_block = "10.101.0.0/24"
}

resource "aws_security_group" "test1" {
    name   = "test1"
    vpc_id = "${aws_vpc.test.id}"
}


resource "aws_security_group" "test2" {
    name   = "test2"
    vpc_id = "${aws_vpc.test.id}"
}

resource "aws_security_group" "test3" {
    name   = "test3"
    vpc_id = "${aws_vpc.test.id}"
}

resource "aws_security_group_rule" "http-in" {
    type      = "ingress"
    from_port = 80
    to_port   = 80
    protocol  = "tcp"
    security_group_id = "${aws_security_group.test1.id}"
    source_security_group_id = "${aws_security_group.test2.id}"
}

resource "aws_security_group_rule" "http-in-two" {
    type      = "ingress"
    from_port = 80
    to_port   = 80
    protocol  = "tcp"
    security_group_id = "${aws_security_group.test1.id}"
    source_security_group_id = "${aws_security_group.test3.id}"
}

resource "aws_security_group_rule" "http-out" {
    type      = "egress"
    from_port = 80
    to_port   = 80
    protocol  = "tcp"
    security_group_id = "${aws_security_group.test2.id}"
    source_security_group_id = "${aws_security_group.test1.id}"
}

Initial plan:

+ aws_security_group.test1
    description: "" => "Managed by Terraform"
    egress.#:    "" => "<computed>"
    ingress.#:   "" => "<computed>"
    name:        "" => "test1"
    owner_id:    "" => "<computed>"
    vpc_id:      "" => "${aws_vpc.test.id}"

+ aws_security_group.test2
    description: "" => "Managed by Terraform"
    egress.#:    "" => "<computed>"
    ingress.#:   "" => "<computed>"
    name:        "" => "test2"
    owner_id:    "" => "<computed>"
    vpc_id:      "" => "${aws_vpc.test.id}"

+ aws_security_group.test3
    description: "" => "Managed by Terraform"
    egress.#:    "" => "<computed>"
    ingress.#:   "" => "<computed>"
    name:        "" => "test3"
    owner_id:    "" => "<computed>"
    vpc_id:      "" => "${aws_vpc.test.id}"

+ aws_security_group_rule.http-in
    from_port:                "" => "80"
    protocol:                 "" => "tcp"
    security_group_id:        "" => "${aws_security_group.test1.id}"
    self:                     "" => "0"
    source_security_group_id: "" => "${aws_security_group.test2.id}"
    to_port:                  "" => "80"
    type:                     "" => "ingress"

+ aws_security_group_rule.http-in-two
    from_port:                "" => "80"
    protocol:                 "" => "tcp"
    security_group_id:        "" => "${aws_security_group.test1.id}"
    self:                     "" => "0"
    source_security_group_id: "" => "${aws_security_group.test3.id}"
    to_port:                  "" => "80"
    type:                     "" => "ingress"

+ aws_security_group_rule.http-out
    from_port:                "" => "80"
    protocol:                 "" => "tcp"
    security_group_id:        "" => "${aws_security_group.test2.id}"
    self:                     "" => "0"
    source_security_group_id: "" => "${aws_security_group.test1.id}"
    to_port:                  "" => "80"
    type:                     "" => "egress"

+ aws_vpc.test
    cidr_block:                "" => "10.101.0.0/24"
    default_network_acl_id:    "" => "<computed>"
    default_security_group_id: "" => "<computed>"
    dhcp_options_id:           "" => "<computed>"
    enable_dns_hostnames:      "" => "<computed>"
    enable_dns_support:        "" => "<computed>"
    main_route_table_id:       "" => "<computed>"

Initial application:

aws_vpc.test: Creating...
  cidr_block:                "" => "10.101.0.0/24"
  default_network_acl_id:    "" => "<computed>"
  default_security_group_id: "" => "<computed>"
  dhcp_options_id:           "" => "<computed>"
  enable_dns_hostnames:      "" => "<computed>"
  enable_dns_support:        "" => "<computed>"
  main_route_table_id:       "" => "<computed>"
aws_vpc.test: Creation complete
aws_security_group.test1: Creating...
  description: "" => "Managed by Terraform"
  egress.#:    "" => "<computed>"
  ingress.#:   "" => "<computed>"
  name:        "" => "test1"
  owner_id:    "" => "<computed>"
  vpc_id:      "" => "vpc-834bf1e6"
aws_security_group.test2: Creating...
  description: "" => "Managed by Terraform"
  egress.#:    "" => "<computed>"
  ingress.#:   "" => "<computed>"
  name:        "" => "test2"
  owner_id:    "" => "<computed>"
  vpc_id:      "" => "vpc-834bf1e6"
aws_security_group.test3: Creating...
  description: "" => "Managed by Terraform"
  egress.#:    "" => "<computed>"
  ingress.#:   "" => "<computed>"
  name:        "" => "test3"
  owner_id:    "" => "<computed>"
  vpc_id:      "" => "vpc-834bf1e6"
aws_security_group.test1: Creation complete
aws_security_group.test2: Creation complete
aws_security_group_rule.http-out: Creating...
  from_port:                "" => "80"
  protocol:                 "" => "tcp"
  security_group_id:        "" => "sg-105c6375"
  self:                     "" => "0"
  source_security_group_id: "" => "sg-115c6374"
  to_port:                  "" => "80"
  type:                     "" => "egress"
aws_security_group_rule.http-in: Creating...
  from_port:                "" => "80"
  protocol:                 "" => "tcp"
  security_group_id:        "" => "sg-115c6374"
  self:                     "" => "0"
  source_security_group_id: "" => "sg-105c6375"
  to_port:                  "" => "80"
  type:                     "" => "ingress"
aws_security_group.test3: Creation complete
aws_security_group_rule.http-in-two: Creating...
  from_port:                "" => "80"
  protocol:                 "" => "tcp"
  security_group_id:        "" => "sg-115c6374"
  self:                     "" => "0"
  source_security_group_id: "" => "sg-135c6376"
  to_port:                  "" => "80"
  type:                     "" => "ingress"
aws_security_group_rule.http-out: Creation complete
aws_security_group_rule.http-in-two: Creation complete
aws_security_group_rule.http-in: Creation complete

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

Subsequent run of plan:

aws_vpc.test: Refreshing state... (ID: vpc-834bf1e6)
aws_security_group.test2: Refreshing state... (ID: sg-105c6375)
aws_security_group.test3: Refreshing state... (ID: sg-135c6376)
aws_security_group.test1: Refreshing state... (ID: sg-115c6374)
aws_security_group_rule.http-out: Refreshing state... (ID: sg-950145865)
aws_security_group_rule.http-in: Refreshing state... (ID: sg-2319652960)
aws_security_group_rule.http-in-two: Refreshing state... (ID: sg-2319652960)

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed.

Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.

-/+ aws_security_group_rule.http-in-two
    from_port:                "80" => "80"
    protocol:                 "tcp" => "tcp"
    security_group_id:        "sg-115c6374" => "sg-115c6374"
    self:                     "false" => "0"
    source_security_group_id: "sg-105c6375" => "sg-135c6376" (forces new resource)
    to_port:                  "80" => "80"
    type:                     "ingress" => "ingress"

[jtopper]

Additional: on the first apply, the tfstate file looks like this (not the same run as above, so the IDs will differ):

                "aws_security_group_rule.http-in": {
                    "type": "aws_security_group_rule",
                    "depends_on": [
                        "aws_security_group.test1",
                        "aws_security_group.test2"
                    ],
                    "primary": {
                        "id": "sg-2319652960",
                        "attributes": {
                            "cidr_blocks.#": "0",
                            "from_port": "80",
                            "id": "sg-2319652960",
                            "protocol": "tcp",
                            "security_group_id": "sg-b4526dd1",
                            "self": "false",
                            "source_security_group_id": "sg-b6526dd3",
                            "to_port": "80",
                            "type": "ingress"
                        }
                    }
                },
                "aws_security_group_rule.http-in-two": {
                    "type": "aws_security_group_rule",
                    "depends_on": [
                        "aws_security_group.test1",
                        "aws_security_group.test3"
                    ],
                    "primary": {
                        "id": "sg-2319652960",
                        "attributes": {
                            "cidr_blocks.#": "0",
                            "from_port": "80",
                            "id": "sg-2319652960",
                            "protocol": "tcp",
                            "security_group_id": "sg-b4526dd1",
                            "self": "false",
                            "source_security_group_id": "sg-b6526dd3",
                            "to_port": "80",
                            "type": "ingress"
                        }
                    }
                },

The http-in-two rule is incorrect here: the source_security_group_id in this case is the ID of the test2 group, whereas I'd expect it to be the test3 group instead.

[jtopper]

I tell a lie, the state file starts out correct after the first apply, but ends up incorrect after the second plan. Something about how the SG rules are read seems to be amiss.

[phinze]

Thanks for the report - we're actively looking into this and we'll follow up later today.

[phinze]

I'm pretty sure this is a scenario that will be fixed by #2376, especially since the key phrase here is "differ only by source security group id". So stay tuned on that PR for the incoming potential fix.

[jtopper]

I've just checked out the ref from #2376 and can confirm my test case now works as expected under that version. Thanks!

[phinze]

Great news!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.