Terraform-Provider-Google uses custom tokenSource logic that does not support OAuth over mTLS

[andyrzhao]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v1.5.0-dev
on linux_amd64

• provider registry.terraform.io/hashicorp/google v4.59.0

Affected Resource(s)

google_*

Terraform Configuration Files

module "pubsub_example_simple" {
source = "../terraform-google-pubsub/examples/simple"
project_id="kaapav-emulator"
}

Expected Behavior

Running terraform apply should successfully create pubsub topic.

Actual Behavior

Running terraform apply results in 403 forbidden in OAuth token exchange step.

Steps to Reproduce

1. terraform apply

Important Factoids

TL:DR This issue only impacts users who are a part of a CAA-policy group that requires all access to GCP APIs to have mTLS enabled, including oauth2 token exchange API. (This group is currently a small subset of GCP developers working on CAA-policy rollout at Google)

Backaground:
The Certificate Based Access team is working on rolling out a CAA policy for Googlers (starting with a subset of gcloud users) that would require them to use mTLS-enabled client tools to access all GCP APIs - this includes the OAuth2 token exchange API (a new requirement). The original mTLS-support for resource APIs was added to terraform ~2 years ago via the transport stack of google-api-go-client. However, the OAuth2 stack uses a separate HTTPClient (configurable via context key oauth2.HTTPClient), which does not have mTLS-support enabled. I authored PR 1886 to fix this problem for google-api-go-client, expecting this to fix the issue for terraform as well. However, it turns out that terraform injects a custom TokenSource and Credentials logic that effectively overrides the mTLS-enabled TokenSource logic provided by google-api-go-client. And upon further digging, it appears that terraform has a duplicate version of the credentials and ADC logic (application default credentials) .

Proposed Solution:
Remove the custom TokenSource logic in Terraform-google-provider and use default TokenSource logic provided by google-api-go-client. Since both code-paths are Google-specific credentials logic, they should be consolidated to not only fix this mTLS bug, but also to provide a consistent user experience w.r.t. supported credential types and order of priority. Steps:

1. Identify gaps between TokenSource logic in terraform-google-provider vs google-api-go-client
2. Implement additional credentials support in google-api-go-client (if any)
3. Update config.go and framework_config.go in Terraform-google-provider to remove custom TokenSource logic when creating HttpClient.

References

• Google-Api-Go-Client PR 1886

[rileykarson]

Fixed by GoogleCloudPlatform/magic-modules#7875

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.