[Q3] Add retry logic to rancher2_cluster update

[a-blender]

Issue: #1040

Problem

Error when upgrading rke cluster with Terraform 1.25

Error: Bad response statusCode [500]. Status [500 Internal Server Error]. Body: [code=ServerError, message=Operation cannot be fulfilled on clusters.management.cattle.io "c-mhjrs": the object has been modified; please apply your changes to the latest version and try again, baseType=error]

which in this case, indicates an intermittent race condition on update of the rancher2_cluster resource.

Solution

Add retry code to rancher2_cluster resource Update. I have chosen to use resource.Retry instead of context.WithTimeout like in this code because we need a cluster update to retry until a non retryable error is returned or the timeout is reached. WithTimeout controls the timeout of an HTTP response, say a 500, but we don't want the request cancelled if it fails. We want to retry several times, especially on upgrade since customers tend to get pretty frustrated if they have to run a terraform apply more than once.

I've also moved updateClusterMonitoring into a helper function to increase readability of the Update function and updated comments.

Testing

Engineering Testing

Manual Testing

The race condition in the original issue is difficult to reproduce, but team discussion has concluded that adding retry code to rancher2_cluster Update will fix the failure on server error.

Test steps

• Provision rke cluster on EC2 nodes with 1 cp/1 etcd/1 worker
• Verify cluster provisions after one tf apply / no HTTP 500 error
• Update rke cluster to increase worker count to 3 + add a fourth node pool
• Verify cluster updates after one tf apply / no HTTP 500 error
• Update rke cluster to enable_cluster_monitoring = true (preferably while the previous update is still running to simulate the race condition)
• Verify cluster updates after one terraform apply (it took 27m for me!) / no HTTP 500 error

main.tf

terraform {
  required_providers {
    rancher2 = {
      source  = "terraform.local/local/rancher2"
      version = "1.0.0"
    }
  }
}
provider "rancher2" {
  api_url   = var.rancher_api_url
  token_key = var.rancher_admin_bearer_token
  insecure  = true
}
data "rancher2_cloud_credential" "rancher2_cloud_credential" {
  name = var.cloud_credential_name
}
resource "rancher2_cluster" "rancher2_cluster" {
  name = var.rke_cluster_name
  rke_config {
    kubernetes_version = "v1.26.4-rancher2-1"
    network {
      plugin = var.rke_network_plugin
    }
  }
}
resource "rancher2_node_template" "rancher2_node_template" {
  name = var.rke_node_template_name
  amazonec2_config {
    access_key           = var.aws_access_key
	  secret_key     = var.aws_secret_key
	  region         = var.aws_region
          ami            = var.aws_ami
	  security_group = [var.aws_security_group_name]
	  subnet_id      = var.aws_subnet_id
	  vpc_id         = var.aws_vpc_id
	  zone           = var.aws_zone_letter
	  root_size      = var.aws_root_size
	  instance_type  = var.aws_instance_type
  }
}
resource "rancher2_node_pool" "pool1" {
  cluster_id       = rancher2_cluster.rancher2_cluster.id
  name             = "pool1"
  hostname_prefix  = "tf-pool1-"
  node_template_id = rancher2_node_template.rancher2_node_template.id
  quantity         = 1
  control_plane    = false
  etcd             = true 
  worker           = false 
}
resource "rancher2_node_pool" "pool2" {
  cluster_id       = rancher2_cluster.rancher2_cluster.id
  name             = "pool2"
  hostname_prefix  = "tf-pool2-"
  node_template_id = rancher2_node_template.rancher2_node_template.id
  quantity         = 1
  control_plane    = true
  etcd             = false 
  worker           = false 
}
resource "rancher2_node_pool" "pool3" {
  cluster_id       = rancher2_cluster.rancher2_cluster.id
  name             = "pool3"
  hostname_prefix  = "tf-pool3-"
  node_template_id = rancher2_node_template.rancher2_node_template.id
  quantity         = 1
  control_plane    = false
  etcd             = false 
  worker           = true 
}

main.tf (sections with cluster updates)

resource "rancher2_cluster" "rancher2_cluster" {
  name = var.rke_cluster_name
  rke_config {
    kubernetes_version = "v1.26.4-rancher2-1"
    network {
      plugin = var.rke_network_plugin
    }
  }
  enable_cluster_monitoring = true
}
.
.
.
resource "rancher2_node_pool" "pool3" {
  cluster_id       = rancher2_cluster.rancher2_cluster.id
  name             = "pool3"
  hostname_prefix  = "tf-pool3-"
  node_template_id = rancher2_node_template.rancher2_node_template.id
  quantity         = 3
  control_plane    = false
  etcd             = false 
  worker           = true 
}
resource "rancher2_node_pool" "pool4" {
  cluster_id       = rancher2_cluster.rancher2_cluster.id
  name             = "pool4"
  hostname_prefix  = "tf-pool4-"
  node_template_id = rancher2_node_template.rancher2_node_template.id
  quantity         = 1
  control_plane    = false
  etcd             = true
  worker           = true
}

Automated Testing

QA Testing Considerations

Regressions Considerations

[HarrisonWAffel]

One small nit about not reusing a client, but otherwise lgtm