Failures updating google_bigquery_data_transfer_config resource when service_account_name is not specified

[SarahFrench]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

We are getting the same error message. This is happening when the provider is upgraded to 4.84.0. If the 4.84.0 is used initially (not from upgrade), there is no issue.

The error happened with or without explicitly setting value for the service_account_name .

See #16033 (comment)

Affected Resource(s)

• google_bigquery_data_transfer_config

Terraform Configuration Files

data "google_project" "project" {}

resource "google_service_account" "bqwriter" {
  account_id = "bqwriter"
}

resource "google_project_iam_member" "data_editor" {
  project = data.google_project.project.project_id

  role   = "roles/bigquery.dataEditor"
  member = "serviceAccount:${google_service_account.bqwriter.email}"
}

resource "google_bigquery_dataset" "my_dataset" {
  dataset_id    = "my_dataset"
  friendly_name = "foo"
  description   = "bar"
  location      = "asia-northeast1"
}

resource "google_bigquery_data_transfer_config" "query_config" {
  depends_on = [google_project_iam_member.data_editor]

  display_name           = "my-query"
  location                     = "asia-northeast1"
  data_source_id         = "scheduled_query"
  schedule                 = "every 15 minutes"
  destination_dataset_id    = google_bigquery_dataset.my_dataset.dataset_id
  service_account_name   = google_service_account.bqwriter.email # remove this field from config and apply an update to the resource
  params = {
    destination_table_name_template = "my_table"
    write_disposition             = "WRITE_APPEND"
    query                         = "SELECT 1 AS a"
  }
}

Debug Output

Panic Output

Expected Behavior

The google_bigquery_data_transfer_config resource is updated successfully and no longer uses the service account previously specified in the config

Actual Behavior

There's an error, which seems to be due to an empty service_account_name value being sent to the API

   Error: Error updating Config "projects/594424405950/locations/asia-northeast1/transferConfigs/6514c398-0000-2c7f-9fe7-883d24fe0948": googleapi: Error 400: The requested service account name () doesn't exist.

Steps to Reproduce

1. terraform apply to create the resource above
2. remove service_account_name from the config and run terraform apply to update the resources

Important Factoids

References

• Failing test : Failing test(s): TestAccBigqueryDataTransferConfig #16033
• Possible change leading to the bug : fix: add updateMask to bqtransfer config updateurl #15359
• Info from user : Failing test(s): TestAccBigqueryDataTransferConfig #16033 (comment)

b/303808567

[SarahFrench]

Note: I'm filing this bug report because the issue for a failing acceptance test was attracting comments outside the usual bug triage process

[MDIDELES]

Thanks @SarahFrench. I am following this because this has big impact to us.

[connor-overjet]

Just wanted to report that we are seeing this in 4.79.0 - We attempted to update an existing Transfer Config on 10/4 first created on 9/12/2023 and saw this same 400 error. Deleting and re-creating the config is our current work around.

[dogmatic69]

Also started getting this error recently.

If I delete the transfer config the next apply works just fine. It's only when trying to do an update that this error starts.

Edit:
Done some more digging and turns out I was defaulting the value to null. When using "" it no longer has this issue.

This has not been the case for the past year or more. Should null be treated as ""?

[dmaltsiniotis-alaska]

Our apply's started failing with this as well.

I believe I have tracked down the reason for this error and it is due to this PR release in version 4.77.0: fix: add updateMask to bqtransfer config updateurl #15359.

The issue with this bugfix is that "serviceAccount" is explicitly hard-coded as one of the fields to update in the updateMask property. However, in the default case where no explicit service account is set, the property is set on the query string as "serviceAccount=" (meaning no/blank value). The PATCH method from the Google REST API (PATCH https://bigquerydatatransfer.googleapis.com/v1/{transferConfig.name=projects/*/locations/*/transferConfigs/*}) seems to interpret this as: "You're telling me to update the serviceAccount property, and you've supplied a serviceAccount query parameter, however it is empty, so you must be mistaken. Here's a a 400."

{
  "error": {
    "code": 400,
    "message": "Request contains an invalid argument.",
    "status": "INVALID_ARGUMENT"
  }
}

Work-around: Revert to provider version 4.76.0 if you are able to and need a solution right now. I've tested this version and it does indeed work for updating transfer configs again. However, there is a minor state file incompatibility between 4.84.0 and 4.76.0 if you're using BigQuery tables (which, presumably you are if you're using data transfer configurations):

│ Warning: Failed to decode resource from state
│
│ Error decoding "google_bigquery_table.table_name" from prior state: unsupported attribute "max_staleness"

Based on the GCP magic-module reference above, it appears Google are preparing to revert this change back to the original behavior, so a fix may be coming soon.

Cheers,

-Demetri

[SarahFrench]

This problem should be fixed by this change, which will be included in the next release : GoogleCloudPlatform/magic-modules#9212

[dogmatic69]

It is still not working in 5.2 (assuming this is the "next change" referenced). Regardless of resources being recreated in 5.2 they still exhibit this error when updating.

Error: Error updating Config "projects/xxx/locations/europe/transferConfigs/xxx": googleapi: Error 400: The requested service account name () doesn't exist.

with module.transfer["xxx"].google_bigquery_data_transfer_config.transfer_config,
  on .terraform/modules/transfer/modules/bigquery/data-transfer/resources.tf line 1, in resource "google_bigquery_data_transfer_config" "transfer_config":
   1: resource "google_bigquery_data_transfer_config" "transfer_config" {

lock file

provider "registry.terraform.io/hashicorp/google" {
  version     = "5.2.0"
  constraints = "< 6.0.0"
  hashes = [
    "h1:8rOD/EyLy8Vefu/fyk8WGA9/4dmGtQGZ+XmqzA0CCmY=",
    "h1:GuKgYrg5q36jxuqrntYHEhnCSsHoNm0tb1af8x8+WLc=",
    "h1:JqmEbXI/bmbzKQUw7Ug2Ue8SsaoPeynhVhuVwXrIiM4=",
    "h1:TCDQMe2CCPoszWoXFibJzNnqwVhoafIRaXysTz4p/xQ=",
    "h1:a05QmpChzywdIrm6tzsl8kKodKlwSqOEapREzlenIfQ=",
    "h1:awmAb6jQ+WJ6TDKuWJuCJSvIluRPAGb9LdfqMD7obno=",
    "h1:kx26+R6hBRzlrDJholsWKob9YrwNI28sOxazE8R5dq0=",
    "h1:psy0RRnGgKCsDKjdXCxQMKt4A1BlcbspWLB5UZK3A5U=",
    "h1:tRRu6twx3B7fN2vXwwY6ODVIajy0hB/wxlwKhubEwF4=",
    "h1:ttjoirB1O0KWdKXMrLRB1SF0uOH6b8CdUbBGxBW2a40=",
    "h1:ufoKADvUrhVrLCzyekI7sq6uMnrh6QKS88nJIDUcyWY=",
  ]
}