Cloud Run metadata annotations: Provider produced inconsistent final plan

[aarcarons]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v1.3.7
on darwin_amd64
+ provider registry.terraform.io/hashicorp/google v4.50.0
+ provider registry.terraform.io/hashicorp/google-beta v4.50.0
+ provider registry.terraform.io/kreuzwerker/docker v2.23.0

Affected Resource(s)

• google_cloud_run_service

Terraform Configuration Files

// Gets information from this Cloud Run service
// if it already exists
data "google_cloud_run_service" "existing" {
  project  = var.project_id
  location = var.region
  name     = var.name
}
locals {
  service  = data.google_cloud_run_service.existing

  // current service metadata. empty list if service hasn't been created yet.
  existing-metadata = coalesce(local.service.metadata, [])

  // current annotations for the service. empty map if no there are no annotations
  existing-annotations = length(local.existing-metadata) == 0 ? {} : lookup(local.existing-metadata[0], "annotations", {})

  // annotation that controls the ingress settins as described in
  // https://cloud.google.com/run/docs/securing/ingress
  ingress-annotation = { "run.googleapis.com/ingress" = var.ingress }

  // the desired annotations for the service
  annotations = merge(local.existing-annotations, local.ingress-annotation)
}
resource "google_cloud_run_service" "service" {
  provider = google-beta
  project  = var.project_id
  location = var.region
  name     = var.name
  metadata {
    annotations = local.annotations
  }
  autogenerate_revision_name = true
  template {
    spec {
      containers {
        image = "..."
        ports {
          name           = "http1"
          container_port = 8080
        }
        resources {
          limits = var.resource_limits
        }

        dynamic "env" {
          for_each = var.cloud_run_environment
          content {
            name  = upper(env.key)
            value = env.value
          }
        }
      }
      container_concurrency = var.container_concurrency
      service_account_name  = local.sa_email
      timeout_seconds       = var.timeout_seconds
    }
    metadata {
      annotations = var.annotations
      labels      = var.labels
    }
  }

  dynamic "traffic" {
    for_each = module.cloud_run_traffic.traffic
    content {
      revision_name   = lookup(traffic.value, "revision_name")
      percent         = lookup(traffic.value, "percent")
      latest_revision = lookup(traffic.value, "latest_revision", null)
    }
  }
}

where var.ingress is a string variable defined as follows:

variable "ingress" {
  description = "Sets the ingress traffic sources allowed to call the service. Allowed values: all, internal or internal-and-cloud-load-balancing"
  type        = string
  default     = "internal"
}

The input given for the failing case is all.

Debug Output

This seems to be a flaky issue. It doesn't always reproduce. When I tried to run the apply with the debug option the issue didn't reproduce.

In the meantime, please see the output of a failing plan - apply.

Plan

  # module.cloud_run.google_cloud_run_service.service will be updated in-place
  ~ resource "google_cloud_run_service" "service" {
        id                         = "..."
        name                       = "..."
        # (4 unchanged attributes hidden)

      ~ metadata {
          ~ annotations      = {
              - "client.knative.dev/user-image"     = "..." -> null
              - "rollout.cloud.run/stableRevision"  = "..." -> null
              - "run.googleapis.com/client-name"    = "..." -> null
              - "run.googleapis.com/client-version" = "..." -> null
              - "run.googleapis.com/ingress"        = "all" -> null
              - "run.googleapis.com/operation-id"   = "..." -> null
                # (3 unchanged elements hidden)
            }
            # (6 unchanged attributes hidden)
        }

      ~ template {

          ~ spec {
                # (3 unchanged attributes hidden)

              ~ containers {
                  ~ image   = "..." -> (known after apply)
                    # (2 unchanged attributes hidden)

                    # (11 unchanged blocks hidden)
                }
            }

            # (1 unchanged block hidden)
        }

      ~ traffic {
          ~ percent         = 100 -> (known after apply)
            # (1 unchanged attribute hidden)
        }

Apply

╷
│ Error: Provider produced inconsistent final plan
│
│ When expanding the plan for module.cloud_run.google_cloud_run_service.service to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/google-beta" produced an invalid new
│ value for .metadata[0].annotations: new element "client.knative.dev/user-image" has appeared.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│
│ When expanding the plan for module.cloud_run.google_cloud_run_service.service to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/google-beta" produced an invalid new
│ value for .metadata[0].annotations: new element "rollout.cloud.run/stableRevision" has appeared.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│
│ When expanding the plan for module.cloud_run.google_cloud_run_service.service to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/google-beta" produced an invalid new
│ value for .metadata[0].annotations: new element "run.googleapis.com/client-name" has appeared.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│
│ When expanding the plan for module.cloud_run.google_cloud_run_service.service to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/google-beta" produced an invalid new
│ value for .metadata[0].annotations: new element "run.googleapis.com/client-version" has appeared.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│
│ When expanding the plan for module.cloud_run.google_cloud_run_service.service to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/google-beta" produced an invalid new
│ value for .metadata[0].annotations: new element "run.googleapis.com/ingress" has appeared.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│
│ When expanding the plan for module.cloud_run.google_cloud_run_service.service to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/google-beta" produced an invalid new
│ value for .metadata[0].annotations: new element "run.googleapis.com/operation-id" has appeared.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Expected Behavior

The purpose of our Terraform code is to add the metadata annotation run.googleapis.com/ingress while keeping any other metadata annotations that the service might have.

In this case, the annotation already exists with the desired value.

The expected behaviour would be that the plan reports no changes to be done on the annotation list. Apply shouldn't be removing any annotation from the list.

Actual Behavior

Plan reports 6 annotations to be removed. Apply fails reporting inconsistent final plan with the error messages shown above.

This is not reproduced 100% of the times.

Steps to Reproduce

With an existing cloud run service and the terraform code shown above.

1. terraform apply

References

• b/272367163

[edwardmedia]

@aarcarons have you checked lifecycle to see if it helps?

[aarcarons]

@edwardmedia I can confirm that after adding an ignore_changes block for each of the unmodified annotations, the plan looks as expected and the apply succeeds.

However, this defeats the purpose of the terraform code above. The goal was to patch the annotations by adding (or updating) the one that controls the ingress settings while also keeping the rest. See the annotations = merge(local.existing-annotations, local.ingress-annotation) part.

Handpicking in advance which annotations need to be excluded and adding them to the ignore_changes block might not always be possible. New annotations might be created independently from terraform, which would lead to the plan failing

[edwardmedia]

@aarcarons Did you notice the behavior of annotation mentioned in the doc? Does that help explain?

[aarcarons]

The Cloud Run API may add additional annotations that were not provided in your config

I understand that additional annotations might be added externally. What I'm looking for is a way to avoid having to ignore these annotations explicitly via lifecycle.ignore_changes. That's why we built the code above: to merge one annotation into the existing list of annotations.

We're passing the full list of annotations every time and still, the diff shows that annotations will be deleted. So there must be something wrong with the provider that causes the inconsistent diff at the plan time. This is actually confirmed by terraform itself at apply time: Error: Provider produced inconsistent final plan.

[edwardmedia]

@aarcarons not sure what causes he diff shows that annotations will be deleted. Are you able to create a simple and static config that can be used to repro the issue?

[aarcarons]

@edwardmedia @melinath I'm not sure how I can create a static example to reproduce this, I don't have much experience with unit testing terraform providers.

I tried to create a smaller example (which would still require to create a cloud run service) but couldn't reproduce. It could depend on the order in which Cloud Run API returns the annotations? Or maybe it's related with the code that suppresses certain annotations from the diff?

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.