RBAC error when following Quick Start

[hbelmiro]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

An error happens when following https://argoproj.github.io/argo-workflows/quick-start/.

Steps to reproduce

Install Argo Workflows

kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.5.2/install.yaml

Patch argo-server authentication

kubectl patch deployment \
  argo-server \
  --namespace argo \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/args", "value": [
  "server",
  "--auth-mode=server"
]}]'

Port-forward the UI

kubectl -n argo port-forward deployment/argo-server 2746:2746

Installed Argo CLI

brew install argo

Submit an example workflow (CLI)

argo submit -n argo --watch https://raw.githubusercontent.com/argoproj/argo-workflows/main/examples/hello-world.yaml

Error

Name:                hello-world-kbmbk
Namespace:           argo
ServiceAccount:      unset (will run with the default ServiceAccount)
Status:              Error
Message:             Error (exit code 1): pods "hello-world-kbmbk" is forbidden: User "system:serviceaccount:argo:default" cannot patch resource "pods" in API group "" in the namespace "argo"
Conditions:          
 PodRunning          False
 Completed           True
Created:             Fri Dec 22 15:20:29 -0300 (10 seconds ago)
Started:             Fri Dec 22 15:20:29 -0300 (10 seconds ago)
Finished:            Fri Dec 22 15:20:39 -0300 (now)
Duration:            10 seconds
Progress:            0/1
ResourcesDuration:   3s*(1 cpu),3s*(100Mi memory)

STEP                  TEMPLATE  PODNAME            DURATION  MESSAGE
 ⚠ hello-world-kbmbk  whalesay  hello-world-kbmbk  5s        Error (exit code 1): pods "hello-world-kbmbk" is forbidden: User "system:serviceaccount:argo:default" cannot patch resource "pods" in API group "" in the namespace "argo"

Version

v3.5.2

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

https://raw.githubusercontent.com/argoproj/argo-workflows/main/examples/hello-world.yaml

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
  labels:
    workflows.argoproj.io/archive-strategy: "false"
  annotations:
    workflows.argoproj.io/description: |
      This is a simple hello world example.
spec:
  entrypoint: whalesay
  templates:
  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [cowsay]
      args: ["hello world"]

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep hello-world-kbmbk           
time="2023-12-22T18:20:29.447Z" level=info msg="Processing workflow" namespace=argo workflow=hello-world-kbmbk
time="2023-12-22T18:20:29.450Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=hello-world-kbmbk
time="2023-12-22T18:20:29.450Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=hello-world-kbmbk
time="2023-12-22T18:20:29.450Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=hello-world-kbmbk
time="2023-12-22T18:20:29.450Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=hello-world-kbmbk
time="2023-12-22T18:20:29.451Z" level=info msg="Pod node hello-world-kbmbk initialized Pending" namespace=argo workflow=hello-world-kbmbk
time="2023-12-22T18:20:29.477Z" level=info msg="Created pod: hello-world-kbmbk (hello-world-kbmbk)" namespace=argo workflow=hello-world-kbmbk
time="2023-12-22T18:20:29.477Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-kbmbk
time="2023-12-22T18:20:29.477Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-kbmbk
time="2023-12-22T18:20:29.488Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=759861 workflow=hello-world-kbmbk
time="2023-12-22T18:20:39.478Z" level=info msg="Processing workflow" namespace=argo workflow=hello-world-kbmbk
time="2023-12-22T18:20:39.478Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=hello-world-kbmbk
time="2023-12-22T18:20:39.478Z" level=info msg="Pod failed: Error (exit code 1): pods \"hello-world-kbmbk\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\"" displayName=hello-world-kbmbk namespace=argo pod=hello-world-kbmbk templateName=whalesay workflow=hello-world-kbmbk
time="2023-12-22T18:20:39.478Z" level=info msg="node changed" namespace=argo new.message="Error (exit code 1): pods \"hello-world-kbmbk\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\"" new.phase=Error new.progress=0/1 nodeID=hello-world-kbmbk old.message= old.phase=Pending old.progress=0/1 workflow=hello-world-kbmbk
time="2023-12-22T18:20:39.478Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=hello-world-kbmbk
time="2023-12-22T18:20:39.478Z" level=info msg=reconcileAgentPod namespace=argo workflow=hello-world-kbmbk
time="2023-12-22T18:20:39.478Z" level=info msg="Updated phase Running -> Error" namespace=argo workflow=hello-world-kbmbk
time="2023-12-22T18:20:39.478Z" level=info msg="Updated message  -> Error (exit code 1): pods \"hello-world-kbmbk\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\"" namespace=argo workflow=hello-world-kbmbk
time="2023-12-22T18:20:39.478Z" level=info msg="Marking workflow completed" namespace=argo workflow=hello-world-kbmbk
time="2023-12-22T18:20:39.484Z" level=info msg="cleaning up pod" action=deletePod key=argo/hello-world-kbmbk-1340600742-agent/deletePod
time="2023-12-22T18:20:39.488Z" level=info msg="Workflow update successful" namespace=argo phase=Error resourceVersion=759977 workflow=hello-world-kbmbk
time="2023-12-22T18:20:39.494Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/hello-world-kbmbk/labelPodCompleted

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=hello-world-kbmbk,workflow.argoproj.io/phase!=Succeeded
time="2023-12-22T18:20:34.605Z" level=info msg="No output parameters"
time="2023-12-22T18:20:34.605Z" level=info msg="No output artifacts"
time="2023-12-22T18:20:34.612Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:argo:default\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"argo\""
time="2023-12-22T18:20:34.613Z" level=warning msg="Non-transient error: pods \"hello-world-kbmbk\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\""
time="2023-12-22T18:20:34.613Z" level=error msg="executor error: pods \"hello-world-kbmbk\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\""
time="2023-12-22T18:20:34.613Z" level=info msg="Alloc=9254 TotalAlloc=13146 Sys=23141 NumGC=3 Goroutines=8"
time="2023-12-22T18:20:34.614Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io \"hello-world-kbmbk\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"argo\""
time="2023-12-22T18:20:34.615Z" level=warning msg="Non-transient error: pods \"hello-world-kbmbk\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\""
time="2023-12-22T18:20:34.615Z" level=error msg="executor error: pods \"hello-world-kbmbk\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\""
time="2023-12-22T18:20:34.615Z" level=fatal msg="pods \"hello-world-kbmbk\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\""

[Joibel]

This is a duplicate of #12391