Provider-AWS does not support GovCloud Partition IDs

[rsavage-nozominetworks]

What happened?

I have multiple AWS accounts, some in the regular Commercial regions, and some in the Gov Cloud regions. All of my composite resources create as expected in the Commercial regions. As for the ones in the Gov Cloud regions, I am seeing issues where certain AWS resources aren't able to create due to improper partition ids. In this example, I am attempting to create a SNS topic which will alert for relative issues. I am providing my provider and controller configs below.

How can we reproduce it?

Original provider configuration (before any attempts to troubleshoot):

apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"aws.upbound.io/v1beta1","kind":"ProviderConfig","metadata":{"annotations":{},"name":"default"},"spec":{"credentials":{"source":"IRSA"}}}
  finalizers:
  - in-use.crossplane.io
  generation: 1
  name: default
spec:
  credentials:
    source: IRSA
status:
  users: 9

An updated provider configuration (after trying to troubleshoot):

apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"aws.upbound.io/v1beta1","kind":"ProviderConfig","metadata":{"annotations":{},"name":"default"},"spec":{"credentials":{"source":"IRSA"}}}
  finalizers:
  - in-use.crossplane.io
  generation: 2
  name: default
spec:
  credentials:
    source: IRSA
  endpoint:
    partitionId: aws-us-gov
    url:
      dynamic:
        host: amazonaws-us-gov.com
        protocol: https
      type: Dynamic
status:
  users: 9

Here is my current ControllerConfig. Notice: I added the env variables for AWS_REGION and AWS_DEFAULT_REGION as recommend by other related issues.

apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws-us-gov:iam::REDACTED:role/REDACTED-crossplane
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"pkg.crossplane.io/v1alpha1","kind":"ControllerConfig","metadata":{"annotations":{"eks.amazonaws.com/role-arn":"arn:aws-us-gov:iam::REDACTED:role/REDACTED-crossplane"},"name":"aws-config"},"spec":{"podSecurityContext":{"fsGroup":2000}}}
  generation: 5
  name: aws-config
spec:
  args:
  - --debug
  env:
  - name: AWS_REGION
    value: us-gov-east-1
  - name: AWS_DEFAULT_REGION
    value: us-gov-east-1
  podSecurityContext:
    fsGroup: 2000

When I run the controller with DEBUG enabled, I see the follow error during SNS topic creation in Gov Cloud:

2023-08-08T17:00:42Z	DEBUG	events	cannot run refresh: refresh failed: reading SNS Topic (arn:aws:sns:us-gov-east-1:REDACTED:REDACTED-notifications-topic): InvalidParameter: Invalid parameter: TopicArn Reason: A us-gov-east-1 ARN must begin with arn:aws-us-gov, not arn:aws:sns:us-gov-east-1:REDACTED:REDACTED-notifications-topic

So from the logs above, it's clear that the provider is trying to use an unsupported ARN in Gov Cloud due to the incorrect partition id. I'd like to know if there is a way to configure that partition id or is this something the developers need to change to properly support Gov Cloud?

Thanks.

What environment did it happen in?

Environment: AWS GovCloud US-GOV-EAST-1
Crossplane version: 1.13.1
Provider version: v0.37.0

[rsavage-nozominetworks]

Just another update. For the provider config, I even tried setting my "host: " to: sns.us-gov-east-1.amazonaws.com, and still getting the error:

023-08-29T21:19:16Z	DEBUG	events	cannot run refresh: refresh failed: reading SNS Topic (arn:aws:sns:us-gov-east-1:REDACTED:REDACTED): InvalidParameter: Invalid parameter: TopicArn Reason: A us-gov-east-1 ARN must begin with arn:aws-us-gov, not arn:aws:sns:us-gov-east-1:REDACTED:REDACTED

[rsavage-nozominetworks]

It maybe related to SNS alone. Can I get some help with triage please.

[mbbush]

The issue seems to be in the way the provider constructs the external name for certain resources, including sns topics, by building the ARN through string concatenation, at a point in the code where the partition is not currently available at runtime (only the region). Searching for "arn:aws: in https://github.com/upbound/provider-aws/blob/main/config/externalname.go will probably show all the affected resources, or at least most of them.

I'm not sure how best to solve this, but I think it explains why the provider seems to work for some resources but not others.

[yebolenko]

I have opened the similar bug, but related to china region (which uses aws-cn partition):
#881
Any progress on this issue?
As I have stated in my original bug, it seems that you need to manage partitionId according to specific region (GOV, CHINA, global). it seems that in the most of components it's just hardcoded: arn:aws:

[github-actions]

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[github-actions]

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[andrewkcarter]

/fresh

[madisonb]

Is there any plan to improve govcloud support within the official provider? Seems silly that the community provider has better govcloud support than the officially supported provider. As of right now, I have to mix and match providers in order to achieve what I need (like the SNS example above).

[jeanduplessis]

@madisonb Do you have access to AWS GovCloud? Would you be willing to provide a PR to add support for it in the provider?

[feifei-xu]

I cross the similar issue to create Policy using provider IAM. I set the region as cn-northwest-1, also I create providerConfig to set the partition to aws-cn

apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: iam
spec:
  credentials:
    source: PodIdentity
  endpoint:
    partitionId: aws-cn
    signingRegion: cn-northwest-1
    source: Custom
    url:
      type: Dynamic
      dynamic:
        host: amazonaws.com.cn
        protocol: https

but the partition still is aws, not aws-cn.