Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

controller-manager error when using jetstream eventbus in v1.7.5 : Failed to watch *v1.Secret #2452

Closed
khmjp opened this issue Feb 3, 2023 · 1 comment · Fixed by #2453
Closed

controller-manager error when using jetstream eventbus in v1.7.5 : Failed to watch *v1.Secret #2452

khmjp opened this issue Feb 3, 2023 · 1 comment · Fixed by #2453
Labels
bug Something isn't working

Comments

@khmjp
Copy link
Contributor

khmjp commented Feb 3, 2023

Describe the bug
When using jetstream eventbus in v1.7.5, controller-manager error Failed to watch *v1.Secret.

To Reproduce
Steps to reproduce the behavior:

  1. install argo events
    https://argoproj.github.io/argo-events/installation/#cluster-wide-installation

  2. deploy jetstream eventbus

cat <<EOL | kubectl apply -f - -n argo-events
apiVersion: argoproj.io/v1alpha1
kind: EventBus
metadata:
  name: default
spec:
  jetstream:
    version: 2.9.1
EOL
  1. See error
I0203 10:01:03.256434       1 leaderelection.go:258] successfully acquired lease argo-events/argo-events-controller
E0203 10:01:03.840913       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: unknown (get secrets)
{"level":"info","ts":1675418464.4760108,"logger":"argo-events.eventbus-controller","caller":"installer/jetstream.go:545","msg":"cluster node hosts: [*.eventbus-default-js-svc.argo-events.svc.cluster.local]","namespace":"argo-events","eventbus":"default","eventbus":"default"}
E0203 10:01:05.138462       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: unknown (get secrets)
{"level":"info","ts":1675418465.2146535,"logger":"argo-events.eventbus-controller","caller":"installer/jetstream.go:605","msg":"created nats server auth secret successfully","namespace":"argo-events","eventbus":"default","eventbus":"default"}
{"level":"info","ts":1675418465.2202623,"logger":"argo-events.eventbus-controller","caller":"installer/jetstream.go:610","msg":"created nats client auth secret successfully","namespace":"argo-events","eventbus":"default","eventbus":"default"}
{"level":"info","ts":1675418465.2259622,"logger":"argo-events.eventbus-controller","caller":"installer/jetstream.go:681","msg":"created jetstream configmap successfully","namespace":"argo-events","eventbus":"default","eventbus":"default"}
{"level":"info","ts":1675418465.235037,"logger":"argo-events.eventbus-controller","caller":"installer/jetstream.go:168","msg":"created jetstream service successfully","namespace":"argo-events","eventbus":"default","eventbus":"default"}
{"level":"info","ts":1675418465.2512054,"logger":"argo-events.eventbus-controller","caller":"installer/jetstream.go:212","msg":"created jetstream statefulset successfully","namespace":"argo-events","eventbus":"default","eventbus":"default"}
{"level":"info","ts":1675418465.712408,"logger":"argo-events.eventbus-controller","caller":"installer/jetstream.go:545","msg":"cluster node hosts: [*.eventbus-default-js-svc.argo-events.svc.cluster.local]","namespace":"argo-events","eventbus":"default","eventbus":"default"}
{"level":"error","ts":1675418466.382953,"logger":"argo-events.eventbus-controller","caller":"installer/jetstream.go:95","msg":"failed to create jetstream auth secrets","namespace":"argo-events","eventbus":"default","eventbus":"default","error":"failed to create nats server auth secret, err: secrets \"eventbus-default-js-server\" already exists","stacktrace":"github.com/argoproj/argo-events/controllers/eventbus/installer.(*jetStreamInstaller).Install\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/installer/jetstream.go:95\ngithub.com/argoproj/argo-events/controllers/eventbus/installer.Install\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/installer/installer.go:33\ngithub.com/argoproj/argo-events/controllers/eventbus.(*reconciler).reconcile\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/controller.go:96\ngithub.com/argoproj/argo-events/controllers/eventbus.(*reconciler).Reconcile\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/controller.go:56\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1675418466.3855438,"logger":"argo-events.eventbus-controller","caller":"installer/installer.go:35","msg":"installation error","namespace":"argo-events","eventbus":"default","error":"failed to create nats server auth secret, err: secrets \"eventbus-default-js-server\" already exists","stacktrace":"github.com/argoproj/argo-events/controllers/eventbus/installer.Install\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/installer/installer.go:35\ngithub.com/argoproj/argo-events/controllers/eventbus.(*reconciler).reconcile\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/controller.go:96\ngithub.com/argoproj/argo-events/controllers/eventbus.(*reconciler).Reconcile\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/controller.go:56\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1675418466.3855863,"logger":"argo-events.eventbus-controller","caller":"eventbus/controller.go:58","msg":"reconcile error","namespace":"argo-events","eventbus":"default","error":"failed to create nats server auth secret, err: secrets \"eventbus-default-js-server\" already exists","stacktrace":"github.com/argoproj/argo-events/controllers/eventbus.(*reconciler).Reconcile\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/controller.go:58\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1675418466.9781373,"logger":"argo-events.eventbus-controller","caller":"installer/jetstream.go:545","msg":"cluster node hosts: [*.eventbus-default-js-svc.argo-events.svc.cluster.local]","namespace":"argo-events","eventbus":"default","eventbus":"default"}
{"level":"error","ts":1675418467.5700028,"logger":"argo-events.eventbus-controller","caller":"installer/jetstream.go:95","msg":"failed to create jetstream auth secrets","namespace":"argo-events","eventbus":"default","eventbus":"default","error":"failed to create nats server auth secret, err: secrets \"eventbus-default-js-server\" already exists","stacktrace":"github.com/argoproj/argo-events/controllers/eventbus/installer.(*jetStreamInstaller).Install\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/installer/jetstream.go:95\ngithub.com/argoproj/argo-events/controllers/eventbus/installer.Install\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/installer/installer.go:33\ngithub.com/argoproj/argo-events/controllers/eventbus.(*reconciler).reconcile\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/controller.go:96\ngithub.com/argoproj/argo-events/controllers/eventbus.(*reconciler).Reconcile\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/controller.go:56\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1675418467.5702708,"logger":"argo-events.eventbus-controller","caller":"installer/installer.go:35","msg":"installation error","namespace":"argo-events","eventbus":"default","error":"failed to create nats server auth secret, err: secrets \"eventbus-default-js-server\" already exists","stacktrace":"github.com/argoproj/argo-events/controllers/eventbus/installer.Install\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/installer/installer.go:35\ngithub.com/argoproj/argo-events/controllers/eventbus.(*reconciler).reconcile\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/controller.go:96\ngithub.com/argoproj/argo-events/controllers/eventbus.(*reconciler).Reconcile\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/controller.go:56\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1675418467.5713463,"logger":"argo-events.eventbus-controller","caller":"eventbus/controller.go:58","msg":"reconcile error","namespace":"argo-events","eventbus":"default","error":"failed to create nats server auth secret, err: secrets \"eventbus-default-js-server\" already exists","stacktrace":"github.com/argoproj/argo-events/controllers/eventbus.(*reconciler).Reconcile\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/controller.go:58\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
E0203 10:01:07.705003       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: unknown (get secrets)
{"level":"info","ts":1675418467.9067497,"logger":"argo-events.eventbus-controller","caller":"installer/jetstream.go:545","msg":"cluster node hosts: [*.eventbus-default-js-svc.argo-events.svc.cluster.local]","namespace":"argo-events","eventbus":"default","eventbus":"default"}
{"level":"error","ts":1675418468.2139149,"logger":"argo-events.eventbus-controller","caller":"installer/jetstream.go:95","msg":"failed to create jetstream auth secrets","namespace":"argo-events","eventbus":"default","eventbus":"default","error":"failed to create nats server auth secret, err: secrets \"eventbus-default-js-server\" already exists","stacktrace":"github.com/argoproj/argo-events/controllers/eventbus/installer.(*jetStreamInstaller).Install\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/installer/jetstream.go:95\ngithub.com/argoproj/argo-events/controllers/eventbus/installer.Install\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/installer/installer.go:33\ngithub.com/argoproj/argo-events/controllers/eventbus.(*reconciler).reconcile\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/controller.go:96\ngithub.com/argoproj/argo-events/controllers/eventbus.(*reconciler).Reconcile\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/controller.go:56\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1675418468.2140582,"logger":"argo-events.eventbus-controller","caller":"installer/installer.go:35","msg":"installation error","namespace":"argo-events","eventbus":"default","error":"failed to create nats server auth secret, err: secrets \"eventbus-default-js-server\" already exists","stacktrace":"github.com/argoproj/argo-events/controllers/eventbus/installer.Install\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/installer/installer.go:35\ngithub.com/argoproj/argo-events/controllers/eventbus.(*reconciler).reconcile\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/controller.go:96\ngithub.com/argoproj/argo-events/controllers/eventbus.(*reconciler).Reconcile\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/controller.go:56\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1675418468.2141445,"logger":"argo-events.eventbus-controller","caller":"eventbus/controller.go:58","msg":"reconcile error","namespace":"argo-events","eventbus":"default","error":"failed to create nats server auth secret, err: secrets \"eventbus-default-js-server\" already exists","stacktrace":"github.com/argoproj/argo-events/controllers/eventbus.(*reconciler).Reconcile\n\t/home/runner/work/argo-events/argo-events/controllers/eventbus/controller.go:58\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
E0203 10:01:11.427299       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: unknown (get secrets)
E0203 10:01:23.233691       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: unknown (get secrets)
E0203 10:01:36.641014       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: unknown (get secrets)

Expected behavior
I expect no error to occur.

Environment (please complete the following information):

  • Kubernetes: v1.25.3
  • Argo: -
  • Argo Events: v1.7.5

Additional context
When I added the secret watch privilege, this error did not occur.
So this pr might be related. #2411

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.
@khmjp khmjp added the bug Something isn't working label Feb 3, 2023
@whynowy
Copy link
Member

whynowy commented Feb 3, 2023

@khmjp - thanks, will fix it and cut a new release!

@whynowy whynowy mentioned this issue Feb 3, 2023
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: remove the secret watch privilege dependency from js eb ctrler whynowy/argo-events
2 participants
@whynowy @khmjp