GKE: Error 403: The caller does not have permission, forbidden

[ryepesg]

Trying to run rbac-lookup with the --gke flag produces:
Error loading RBAC Bindings: googleapi: Error 403: The caller does not have permission, forbidden

The problem happens with multiples configurations of clusters:

gcloud container clusters create --num-nodes=1 --disk-size=10Gi --machine-type=g1-small --scopes compute-rw,storage-ro,gke-default c1

gcloud container clusters update --enable-basic-auth c1

gcloud container clusters create --num-nodes=1 --disk-size=10Gi --machine-type=g1-small --node-version=1.12.5-gke.10 --cluster-version=1.12.5-gke.10 --scopes compute-rw,storage-ro,gke-default c1

The problem occurs even using a service account with the Owner role:

gcloud container clusters create --num-nodes=1 --disk-size=10Gi --machine-type=g1-small --service-account=$NODE_SA_EMAIL --node-version=1.12.5-gke.10 --cluster-version=1.12.5-gke.10 c1

[sudermanjr]

In GKE, owner does not grant full permissions inside the kubernetes cluster. My guess is that you need to grant your user a clusterrolebinding with higher privileges.

This stack overflow does a decent job of describing the process:
https://stackoverflow.com/questions/46307325/gke-clusterrolebinding-for-cluster-admin-fails-with-permission-error

[ryepesg]

Thank you @sudermanjr, but I get the same error even with the cluster-admin clusterrolebinding assigned to my user:

% kubectl describe clusterrolebindings cluster-admin-binding 
Name:         cluster-admin-binding
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  cluster-admin
Subjects:
  Kind  Name                      Namespace
  ----  ----                      ---------
  User <MY_EMAIL>

[sudermanjr]

Ah, the issue is not the account used to start the cluster, it's the account you are using when you run rbac-lookup and the permissions that it has in your GCP account. Basically, you need to be able to run some form of gcloud iam command in your project. I can dig further to find out exactly what's needed, but that's the direction to look

[ryepesg]

Thank you, but I am using the same account to create the cluster and also to run rbac-lookup (my user is the owner of the project and I am creating a cluster-admin role for it):

gcloud container clusters create kube

kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $(gcloud config get-value account)

rbac-lookup --gke
Error loading RBAC Bindings: googleapi: Error 403: The caller does not have permission, forbidden

[robscott]

Hey @ryepesg, thanks for reporting this problem! I think the confusion here is that you actually need GCP IAM access here, not just GCP cluster access. Even a GKE cluster admin might not actually have project IAM access.

There's 3 calls this could be happening on, to help me rule out one of them, can you try to run this command?

gcloud projects get-iam-policy YOUR_PROJECT_NAME_HERE

If that works for you, that rules out a problem with https://github.com/reactiveops/rbac-lookup/blob/master/lookup/lister.go#L198. On the other hand, if it doesn't you'll need more IAM access for your account.

Either way, there are a couple places above that in that method that could also trigger an error like that. I'll work on adding some better error handling so we can tell which of those actually triggered the error, as well as changing the initial scope request to be read only.

Thanks again for all the help debugging this!

[ryepesg]

Thank you @robscott, I am the owner of the project in IAM and I am also a cluster-admin of the cluster. Here is the output of the command:

% gcloud projects get-iam-policy $(gcloud config get-value project)
bindings:
- members:
  - group:<A_GROUP_EMAIL>
  role: projects/oauthz/roles/CustomRole
- members:
  - serviceAccount:[email protected]
  role: roles/compute.admin
- members:
  - serviceAccount:[email protected]
  role: roles/compute.serviceAgent
- members:
  - user:<MY_EMAIL>
  role: roles/container.admin
- members:
  - serviceAccount:service-140183570641@container-engine-robot.iam.gserviceaccount.com
  role: roles/container.serviceAgent
- members:
  - serviceAccount:[email protected]
  - serviceAccount:[email protected]
  - serviceAccount:[email protected]
  role: roles/editor
- members:
  - serviceAccount:[email protected]
  role: roles/logging.logWriter
- members:
  - serviceAccount:[email protected]
  role: roles/monitoring.metricWriter
- members:
  - serviceAccount:[email protected]
  role: roles/monitoring.viewer
- members:
  - user:<MY_EMAIL>
  role: roles/owner
- members:
  - serviceAccount:[email protected]
  role: roles/storage.objectViewer

Thank you,

[robscott]

@ryepesg I've released a new version of rbac-lookup with one potential fix, along with some more helpful error messages. Let me know if that helps narrow down the problem. Thanks again for all the help!

[ryepesg]

Thank you. Here is my new attempt using a service account:

% kubectl describe clusterrolebinding cluster-admin-binding
Name:         cluster-admin-binding
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  cluster-admin
Subjects:
  Kind  Name                                   Namespace
  ----  ----                                   ---------
  User  [email protected]

% gcloud config get-value project
oauthz
% gcloud config get-value account
[email protected]
% rbac-lookup --gke
Error loading Google Cloud IAM Policy for project: oauthz
Error loading RBAC Bindings: googleapi: Error 403: The caller does not have permission, forbidden

[robscott]

Hey @ryepesg, thanks for all that output! My best guess here is that the logic for parsing the GCP project ID here was not functioning here. When you use the gcloud CLI to get GKE credentials, it generally initializes a cluster with a name following this pattern:

gke_reactiveopsio_us-central1-a_rbac-lookup-testing

I split that by underscores, and the 2nd element in the resulting array generally represents the GCP project name. In your case that computed to oauthz which seems unlikely to be accurate. In the latest release I added a couple fallback methods:

• It will get the GCP application default credentials as configured in your environment and attempt to parse a project ID from those
• It will get the value of the CLOUDSDK_CORE_PROJECT environment variable used by GCP to reference the project ID

In both cases, if there is a value found, it will attempt to use it to load an IAM policy, and if that fails, it will move on to the next one. Hopefully this helps, thanks for all the detailed feedback!

[ryepesg]

Thanks for the answer. This my new run:

% rbac-lookup --gke
Could not load IAM policy for oauthz project from parsed kubeconfig
No project ID found in default GCP credentials
Error loading RBAC Bindings: Error loading IAM policies for GKE, try setting CLOUDSDK_CORE_PROJECT environment variable

% CLOUDSDK_CORE_PROJECT=oauthz rbac-lookup --gke                                                                             :(
Could not load IAM policy for oauthz project from parsed kubeconfig
No project ID found in default GCP credentials
Could not load IAM policy for oauthz project from CLOUDSDK_CORE_PROJECT environment variable
Error loading RBAC Bindings: googleapi: Error 403: The caller does not have permission, forbidden

% grep oauthz ~/.kube/config                                                                                                 :(
  name: gke_oauthz_australia-southeast1-a_kube
    cluster: gke_oauthz_australia-southeast1-a_kube
    user: gke_oauthz_australia-southeast1-a_kube
  name: gke_oauthz_australia-southeast1-a_kube
current-context: gke_oauthz_australia-southeast1-a_kube
- name: gke_oauthz_australia-southeast1-a_kube

Using a different project, with a different user:

% set gcloud config set account <OTHER_EMAIL>
% gcloud container clusters create kube --project platform-test
% rbac-lookup --gke
Could not load IAM policy for platform-test project from parsed kubeconfig
No project ID found in default GCP credentials
Error loading RBAC Bindings: Error loading IAM policies for GKE, try setting CLOUDSDK_CORE_PROJECT environment variable
% grep platform-test ~/.kube/config                                                                                          
  name: gke_platform-test_australia-southeast1-a_kube
    cluster: gke_platform-test_australia-southeast1-a_kube
    user: gke_platform-test_australia-southeast1-a_kube
  name: gke_platform-test_australia-southeast1-a_kube
current-context: gke_platform-test_australia-southeast1-a_kube
- name: gke_platform-test_australia-southeast1-a_kube

% CLOUDSDK_CORE_PROJECT=platform-test rbac-lookup --gke
Could not load IAM policy for platform-test project from parsed kubeconfig
No project ID found in default GCP credentials
Could not load IAM policy for platform-test project from CLOUDSDK_CORE_PROJECT environment variable
Error loading RBAC Bindings: googleapi: Error 403: The caller does not have permission, forbidden

However, the rbac-lookup --gke does work from my office with an existing cluster. Maybe this is a bug just working with new clusters. Will close the ticket, thanks a lot for your help.

[robscott]

@ryepesg Somehow I just realized I was likely going down the wrong, much more complicated path here. What happens if you run gcloud auth application-default login before running rbac-lookup? The Google auth I'm using apparently searches for "application default credentials" which can be different than the credentials used by the gcloud CLI command: https://github.com/golang/oauth2/blob/HEAD/google/default.go#L61

[ryepesg]

Thank you @robscott! That was the problem indeed. I even tested changing the account with gcloud config set account to a different one, but rbac-lookup keep using the application default credentials (created by gcloud auth application-default login in .config/gcloud/application_default_credentials.json).

So, the correct way to change the account is:

GOOGLE_APPLICATION_CREDENTIALS=serviceaccount.json rbac-lookup --gke

That works even after using gcloud auth application-default revoke.

Thanks for all the help, cheers,

[robscott]

Nice! Glad that solved it. I'll work on updating the docs to be more helpful here. Thanks for all your help working through this!