Getting "RBAC: access denied" when connecting with CLI to endpoint that is available in browser

[karlschriek]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I am serving the ArgoCD Server on argocd.mydomain.com using the following Istio setup:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: argocd
  namespace: istio-gateway
spec:
  selector:
    istio: gateway
  servers:
    - hosts:
        - "argocd.mydomain.com"
      port:
        name: http
        number: 80
        protocol: HTTP
      tls:
        httpsRedirect: true #redirect http to https
    - hosts:
        - "argocd.mydomain.com"
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: "argocd-cert"

and:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: argocd-server
  namespace: argocd
spec:
  gateways:
    - istio-gateway/argocd
  hosts:
    - "argocd.mydomain.com"
  http:
    - route:
        - destination:
            host: argocd-server
            port:
              number: 80

I can log into the server via my browser at argocd.mydomain.com. However, if I tried to log in via the CLI, I am hit with an RBAC: access denied, which means that Istio is blocking traffic that hasn't been allowed for via the setup above. The only thing that I can think of is that the CLI might be attempting to use a different port.

$ argocd login argocd.mydomain.com --username admin --password pwdfoo123 --insecure #using --insecure because the TLS certs are non-valid staging certs)
FATA[0000] rpc error: code = PermissionDenied desc = RBAC: access denied

I also attempted to use the suggestions in #2784 (comment):

$ argocd login argocd.mydomain.com --username admin --password pwdfoo123 --insecure --grpc-web
FATA[0002] rpc error: code = Unknown desc = POST https://argocd.mydomain.com:443/session.SessionService/Create failed with status code 403 

Expected behavior

Since the server is correctly serving to argocd.mydomain.com and can be connect to there via the browser, I would expect the CLI to be able to connect on that endpoint.

Version

argocd: v2.4.0+91aefab
  BuildDate: 2022-06-10T17:44:14Z
  GitCommit: 91aefabc5b213a258ddcfe04b8e69bb4a2dd2566
  GitTreeState: clean
  GoVersion: go1.18.3
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.4.0+91aefab
  BuildDate: 2022-06-10T17:23:37Z
  GitCommit: 91aefabc5b213a258ddcfe04b8e69bb4a2dd2566
  GitTreeState: clean
  GoVersion: go1.18.3
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.4.1 2021-11-11T23:36:27Z
  Helm Version: v3.8.1+g5cb9af4
  Kubectl Version: v0.23.1
  Jsonnet Version: v0.18.0

[jessesuen]

Does login work via port-forwarding and running argocd login localhost:XX?

[karlschriek]

No, this doesn't work at all when --insecure is set. (When not set it works without any issues).

Here are the errors. Initial port-forwarding succeeds:

$ kubectl port-forward svc/argocd-server -n argocd 8888:80
W0818 19:56:08.389740  103506 azure.go:92] WARNING: the azure auth plugin is deprecated in v1.22+, unavailable in v1.25+; use https://github.com/Azure/kubelogin instead.
To learn more, consult https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins
Forwarding from 127.0.0.1:8888 -> 8080
Forwarding from [::1]:8888 -> 8080
Handling connection for 8888
Handling connection for 8888

Thereafter, whether attempting to connect via the browser or via the CLI, the following output is printed in the console logs and the port-forwarding is broken:

E0818 19:56:12.735346  103506 portforward.go:406] an error occurred forwarding 8888 -> 8080: error forwarding port 8080 to pod 922c222aa17d861ed872caf98729cf0a18303b5a08ca6e49b3dd161abcdfcd55, uid : failed to execute portforward in network namespace "/var/run/netns/cni-d0ff4986-3c2b-3485-77f5-5c7e9cea95a4": read tcp4 127.0.0.1:38170->127.0.0.1:8080: read: connection reset by peer
E0818 19:56:12.735531  103506 portforward.go:234] lost connection to pod
Handling connection for 8888
E0818 19:56:12.735711  103506 portforward.go:346] error creating error stream for port 8888 -> 8080: EOF

The CLI itself just prints "connection refused":

$ argocd login localhost:8888
FATA[0002] dial tcp 127.0.0.1:8888: connect: connection refused 

EDIT: this particular issue looks like it is related to #8613

[karlschriek]

This gives the exact same error:
argocd login localhost:8888 --plaintext

This does work:
argocd login --port-forward --plaintext --port-forward-namespace argocd

...which is good to know in the situation where I can piggy-back off the kubernetes API, but what I want to is have argocd login argocd.mydomain.com work

[deejiw]

With only --insecure enabled, login on browser (localhost:15032) gives the following errors:

URL: http://localhost:15032/api/v1/applications?fields=metadata.resourceVersion%2Citems.metadata.name%2Citems.metadata.annotations%2Citems.metadata.labels%2Citems.metadata.creationTimestamp%2Citems.metadata.deletionTimestamp%2Citems.spec%2Citems.operation.sync%2Citems.status.sync.status%2Citems.status.health%2Citems.status.operationState.phase%2Citems.status.operationState.operation.sync%2Citems.status.summary&selector=
Status: 401 Unauthorized

URL: http://localhost:15032/api/v1/clusters
Status: 401 Unauthorized

virtual-service.yaml

kind: VirtualService
metadata:
  name: ui-services
spec:
  hosts:
  - "*"
  gateways:
  - local-gateway
  http:
  - name: argocd
    match:
      - port: 15032
    route:
      - destination:
          host: argocd-server.argocd.svc.cluster.local
          port:
            number: 80