google_dataproc_cluster- Internal IPs not preserved for master and worker nodes when redeployed

[sehgalnamit]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

*Terraform 0.13+
terraform {
required_providers {
google = {
source = "hashicorp/google"
version = "3.84.0"
}
}
}

provider "google" {

Configuration options

dataproc_clusters = {
"welltuned" : {
software_config_optional_components = ["ZOOKEEPER", "JUPYTER", "FLINK"]
}
}

Affected Resource(s)

• google_dataproc_cluster

Terraform Configuration Files

# dataproc_clusters = {
  "welltuned" : {
    software_config_optional_components = ["ZOOKEEPER", "JUPYTER", "FLINK"]
  }
}

variable "dataproc_clusters" {
  type    = any
  default = {}
}

locals {
  dataproc_clusters = var.dataproc_clusters
}


# Grant https://cloud.google.com/compute/docs/access/iam#compute.networkUser on SharedVPC 
resource "google_dataproc_cluster" "project_dataproc" {
  for_each = local.dataproc_clusters

  project = local.project_id

  name                          = format("%s-%s", each.key, random_id.project_dataproc_suffix[index(keys(local.dataproc_clusters), each.key)].hex)
  region                        = local.availability_regions[lookup(each.value, "location", "xxxxx")]
  graceful_decommission_timeout = lookup(each.value, "graceful_decommission_timeout", "120s")


  cluster_config {
    staging_bucket = lookup(each.value, "staging_bucket", null)

    master_config {
      num_instances = lookup(each.value, "master_config_num_instances", 1)
      machine_type  = lookup(each.value, "master_config_machine_type", "e2-medium")
      disk_config {
        boot_disk_type    = lookup(each.value, "master_config_boot_disk_type", "pd-ssd")
        boot_disk_size_gb = lookup(each.value, "master_config_boot_disk_size_gb", 30)
      }
    }

    worker_config {
      num_instances    = lookup(each.value, "worker_config_num_instances", 2)
      machine_type     = lookup(each.value, "worker_config_machine_type", "e2-medium")
      min_cpu_platform = lookup(each.value, "worker_config_min_cpu_platform", null)
      disk_config {
        boot_disk_size_gb = lookup(each.value, "worker_config_boot_disk_size_gb", 30)
        num_local_ssds    = lookup(each.value, "worker_config_num_local_ssds", 0)
      }
    }

    preemptible_worker_config {
      num_instances = lookup(each.value, "preemptible_worker_config_num_instances", 0)
    }

    software_config {
      image_version       = lookup(each.value, "software_config_image_version", "2.0-debian10")
      override_properties = lookup(each.value, "software_config_override_properties", {})
      optional_components = lookup(each.value, "software_config_optional_components", [])
    }

    gce_cluster_config {
      tags            = lookup(each.value, "gce_cluster_config_tags", [])
      service_account = local.default_service_account.email
      service_account_scopes = [
        "cloud-platform"
      ]
      internal_ip_only = true
      subnetwork       = local.default_project_subnetworks[lookup(each.value, "location", "az1")].self_link
    }
  }

  depends_on = [google_project_iam_binding.dataproc_shared_vpc_networkuser_role, ]

}

output "dataproc_clusters" {
  value = google_dataproc_cluster.project_dataproc
}

#

Debug Output

Panic Output

Expected Behavior

Master and worker nodes IP remains same

Actual Behavior

With google_dataproc_cluster- Internal IPs not preserved for master and worker nodes when redeployed
tfstate recreates the nester and worker node internal IPS for newly created dataproc lsuter
trace.txt

I have tried manually restarting the dataproc cluster and IPs do not change.
Only when I redeploy using terraform , my tfstate doe snot stop IPs to be preserved and it recreates.

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #0000

[ScottSuarez]

This is how terraform works. If a configuration change can not be done through an update call the resource destroyed and a new resource is created using the config defined within terraform file. There is no way to carry over fields defined in a previous deployment as they are outside of your configuration files.

[sehgalnamit]

@ScottSuarez
Thanks for your reply
I am not sure if you have reproduce or checked my logs at your end.

This is causing the issue
~ service_account_scopes = [ # forces replacement
2021-09-16 09:50:16 - "https://www.googleapis.com/auth/cloud.useraccounts.readonly",
2021-09-16 09:50:16 - "https://www.googleapis.com/auth/devstorage.read_write",
2021-09-16 09:50:16 - "https://www.googleapis.com/auth/logging.write",
2021-09-16 09:50:16 # (1 unchanged element hidden)
2021-09-16 09:50:16 ]

If I read on documentation
service_account_scopes - (Optional, Computed) The set of Google API scopes to be made available on all of the node VMs under the service_account specified. Both OAuth2 URLs and gcloud short names are supported. To allow full access to all Cloud APIs, use the cloud-platform scope. See a complete list of scopes here.

This is in my codes
service_account = local.default_service_account.email
service_account_scopes = [
"cloud-platform"
]

[sehgalnamit]

I think this is the solution for this, I have to hardcode:-
Using gce_cluster_config -> service_account_scopes in google_dataproc_cluster always recreate the cluster · Issue #4403 · hashicorp/terraform-provider-google (github.com)

[sehgalnamit]

service_account_scopes = lookup(each.value, "service_account_scopes", ["https://www.googleapis.com/auth/cloud.useraccounts.readonly", "https://www.googleapis.com/auth/devstorage.read_write", "https://www.googleapis.com/auth/logging.write", "https://www.googleapis.com/auth/sqlservice.admin"])

This should fix the issue.

[sehgalnamit]

This works with no alias for one URL
service_account_scopes = [
"https://www.googleapis.com/auth/cloud.useraccounts.readonly", "storage-rw", "logging-write", "sql-admin"
]

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.