Update ecr-credential-provider to version that supports authenticating public ECR registries

[mmerkes]

What would you like to be added:
Today, the ecr-credential-provider does not support authenticating public ECR registries, but that was recently added to the credential provider. Once added, customers can update their credential provider config to start using it.

Something like this:

{
  "apiVersion": "kubelet.config.k8s.io/v1",
  "kind": "CredentialProviderConfig",
  "providers": [
    {
      "name": "ecr-credential-provider",
      "matchImages": [
        "public.ecr.aws",
        "*.dkr.ecr.*.amazonaws.com",
        "*.dkr.ecr.*.amazonaws.cn",
        "*.dkr.ecr-fips.*.amazonaws.com",
        "*.dkr.ecr.us-iso-east-1.c2s.ic.gov",
        "*.dkr.ecr.us-isob-east-1.sc2s.sgov.gov"
      ],
      "defaultCacheDuration": "12h",
      "apiVersion": "credentialprovider.kubelet.k8s.io/v1"
    }
  ]
}

The nodes will also need additional permissions to make authenticated calls for ecr public:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr-public:GetAuthorizationToken",
                "sts:GetServiceBearerToken"
            ],
            "Resource": "*"
        }
    ]
}

Initially, we won't update the config in this repo because calls will fail for customers that don't have sufficient permissions, which are not given by default in managed roles, and will generate noise and confusion, but once the binaries are published, customers can start updating their own config to use it.

Why is this needed:

Today, customers need to manually build the binary and update their nodes. This allows customers to make authenticate against public ECR registries, which avoid the 500GB unauthenticated call limit.

[jenademoodley]

Testing on my side, It appears that the ecr-credential-provider binary has already been updated on the new AMI releases. I created a node group from the 1.28-v20240506 AMI to test, and it does call ecr-public:GetAuthorizationToken. Created the node group with eksctl and specified the AMI as a custom AMI. eksctl cluster config is as below:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: ca-1-27-test
  region: eu-west-1
  version: "1.28"

managedNodeGroups:
  - name: ng-cred
    instanceType: m5.large
    desiredCapacity: 2
    amiFamily: AmazonLinux2
    ami: ami-033a6b4978ebf28b5
    iam:
      attachPolicy:
        Version: "2012-10-17"
        Statement:
        - Effect: Allow
          Action:
          - 'ecr-public:GetAuthorizationToken'
          - 'sts:GetServiceBearerToken'
          Resource: '*'
    overrideBootstrapCommand: |
      #!/bin/bash
      mv /etc/eks/image-credential-provider/config.json /etc/eks/image-credential-provider/config.json.bak
      cat > /etc/eks/image-credential-provider/config.json << EOF
      {
        "apiVersion": "kubelet.config.k8s.io/v1",
        "kind": "CredentialProviderConfig",
        "providers": [
          {
            "name": "ecr-credential-provider",
            "matchImages": [
              "public.ecr.aws",
              "*.dkr.ecr.*.amazonaws.com",
              "*.dkr.ecr.*.amazonaws.com.cn",
              "*.dkr.ecr-fips.*.amazonaws.com",
              "*.dkr.ecr.*.c2s.ic.gov",
              "*.dkr.ecr.*.sc2s.sgov.gov"
            ],
            "defaultCacheDuration": "12h",
            "apiVersion": "credentialprovider.kubelet.k8s.io/v1"
          }
        ]
      }
      EOF
      /etc/eks/bootstrap.sh ca-1-27-test

[mmerkes]

The ecr-credential-provider with this change has been built in for a while. Resolving.

[jenademoodley]

Tested on my end without specifying the permissions for the node:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: ca-1-27-test
  region: eu-west-1
  version: "1.28"

managedNodeGroups:
  - name: ng-cred
    instanceType: m5.large
    desiredCapacity: 2
    amiFamily: AmazonLinux2
    ami: ami-033a6b4978ebf28b5
    overrideBootstrapCommand: |
      #!/bin/bash
      mv /etc/eks/image-credential-provider/config.json /etc/eks/image-credential-provider/config.json.bak
      cat > /etc/eks/image-credential-provider/config.json << EOF
      {
        "apiVersion": "kubelet.config.k8s.io/v1",
        "kind": "CredentialProviderConfig",
        "providers": [
          {
            "name": "ecr-credential-provider",
            "matchImages": [
              "public.ecr.aws",
              "*.dkr.ecr.*.amazonaws.com",
              "*.dkr.ecr.*.amazonaws.com.cn",
              "*.dkr.ecr-fips.*.amazonaws.com",
              "*.dkr.ecr.*.c2s.ic.gov",
              "*.dkr.ecr.*.sc2s.sgov.gov"
            ],
            "defaultCacheDuration": "12h",
            "apiVersion": "credentialprovider.kubelet.k8s.io/v1"
          }
        ]
      }
      EOF
      /etc/eks/bootstrap.sh ca-1-27-test

The resulting node is still able to pull and run images from ECR gallery, although the authentication action still runs and fails as expected:

[root@ip-192-168-15-234 ~]# journalctl | grep public
Jun 20 14:19:27 ip-192-168-15-234.eu-west-1.compute.internal kubelet[3126]: E0620 14:19:27.593911    3126 plugin.go:233] Failed getting credential from external registry credential provider: error execing credential provider plugin ecr-credential-provider for image public.ecr.aws/xray/aws-xray-daemon: exit status 1: I0620 14:19:27.243671    4323 main.go:94] Getting creds for public registry
Jun 20 14:19:27 ip-192-168-15-234.eu-west-1.compute.internal kubelet[3126]: E0620 14:19:27.591458    4323 main.go:257] Error running credential provider plugin: AccessDeniedException: User:arn:aws:sts::<ACCOUNT_ID>:assumed-role/eksctl-ca-1-27-test-nodegroup-ng-c-NodeInstanceRole-tpCBDDklJWpZ/i-089810440fb31b647 is not authorized to perform: ecr-public:GetAuthorizationToken onresource: * because no identity-based policy allows the ecr-public:GetAuthorizationToken action
Jun 20 14:19:27 ip-192-168-15-234.eu-west-1.compute.internal containerd[2893]: time="2024-06-20T14:19:27.600749872Z" level=info msg="PullImage \"public.ecr.aws/xray/aws-xray-daemon:latest\""
Jun 20 14:19:30 ip-192-168-15-234.eu-west-1.compute.internal containerd[2893]: time="2024-06-20T14:19:30.962826880Z" level=info msg="ImageCreate event name:\"public.ecr.aws/xray/aws-xray-daemon:latest\" labels:{key:\"io.cri-containerd.image\" value:\"managed\"}"
Jun 20 14:19:30 ip-192-168-15-234.eu-west-1.compute.internal containerd[2893]: time="2024-06-20T14:19:30.974266952Z" level=info msg="stop pulling image public.ecr.aws/xray/aws-xray-daemon:latest: active requests=0, bytes read=4166721"
Jun 20 14:19:30 ip-192-168-15-234.eu-west-1.compute.internal containerd[2893]: time="2024-06-20T14:19:30.986836081Z" level=info msg="ImageUpdate event name:\"public.ecr.aws/xray/aws-xray-daemon:latest\" labels:{key:\"io.cri-containerd.image\" value:\"managed\"}"

Based on the above, can we consider if we should add this change to the credential helper, or should we modify/add to documentation stating how it can be used. Reason being is that a few users have experienced rate limits against ECR public gallery due to the unauthenticated pulls and can't request an increase (only authenticated pulls request can be increased)