Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

controller panic after 20 times syncing sensor if user not config Spec.Escalation #86

Closed
fatkun opened this issue Aug 10, 2018 · 1 comment · Fixed by #89
Closed

controller panic after 20 times syncing sensor if user not config Spec.Escalation #86

fatkun opened this issue Aug 10, 2018 · 1 comment · Fixed by #89
Assignees
Labels
bug Something isn't working

Comments

@fatkun
Copy link

fatkun commented Aug 10, 2018

Describe the bug
controller panic after 20 times syncing sensor

To Reproduce
Steps to reproduce the behavior:

  1. stop signal deploy
  2. wait 20 times retry
  3. See error

the error is produce here, It should not sendMessage if user not config Spec.Escalation.

	if err != nil {
		// now let's escalate the sensor
		// the context should have the most up-to-date version
		log.Infof("escalating sensor to level %s via %s message", ctx.s.Spec.Escalation.Level, ctx.s.Spec.Escalation.Message.Stream.Type)
		err := sendMessage(&ctx.s.Spec.Escalation.Message)
		if err != nil {
			log.Panicf("failed escalating sensor '%s'", key)
		}
	}
ERROR: logging before flag.Parse: E0810 07:30:23.167717       1 runtime.go:66] Observed a panic: &logrus.Entry{Logger:(*logrus.Logger)(0xc4203f0140), Data:logrus.Fields{}, Time:time.Time{wall:0xbed36da3c9fcf676, ext:4169274126066, loc:(*time.Location)(0x1fbbf40)}, Level:0x0, Message:"failed escalating sensor 'argo/test2'", Buffer:(*bytes.Buffer)(nil)} (&{0xc4203f0140 map[] 2018-08-10 07:30:23.16757311 +0000 UTC m=+4169.274126066 panic failed escalating sensor 'argo/test2' <nil>})
/Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/Cellar/go/1.10/libexec/src/runtime/asm_amd64.s:573
/usr/local/Cellar/go/1.10/libexec/src/runtime/panic.go:505
/Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/github.com/sirupsen/logrus/entry.go:126
/Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/github.com/sirupsen/logrus/entry.go:194
/Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/github.com/sirupsen/logrus/entry.go:242
/Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/github.com/sirupsen/logrus/logger.go:181
/Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/github.com/sirupsen/logrus/exported.go:155
/Users/mmagaldi/go/src/github.com/argoproj/argo-events/controller/controller.go:125
/Users/mmagaldi/go/src/github.com/argoproj/argo-events/controller/controller.go:182
/Users/mmagaldi/go/src/github.com/argoproj/argo-events/controller/controller.go:175
/Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/Cellar/go/1.10/libexec/src/runtime/asm_amd64.s:2361
panic: (*logrus.Entry) (0x1533b40,0xc4213f0e10) [recovered]
        panic: (*logrus.Entry) (0x1533b40,0xc4213f0e10)

goroutine 88 [running]:
github.com/argoproj/argo-events/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107
panic(0x1533b40, 0xc4213f0e10)
        /usr/local/Cellar/go/1.10/libexec/src/runtime/panic.go:505 +0x229
github.com/argoproj/argo-events/vendor/github.com/sirupsen/logrus.Entry.log(0xc4203f0140, 0xc420959320, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/github.com/sirupsen/logrus/entry.go:126 +0x2d2
github.com/argoproj/argo-events/vendor/github.com/sirupsen/logrus.(*Entry).Panic(0xc4213f0c30, 0xc4210fdce0, 0x1, 0x1)
        /Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/github.com/sirupsen/logrus/entry.go:194 +0xaa
github.com/argoproj/argo-events/vendor/github.com/sirupsen/logrus.(*Entry).Panicf(0xc4213f0c30, 0x157ff71, 0x1d, 0xc4210fdde0, 0x1, 0x1)
        /Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/github.com/sirupsen/logrus/entry.go:242 +0xed
github.com/argoproj/argo-events/vendor/github.com/sirupsen/logrus.(*Logger).Panicf(0xc4203f0140, 0x157ff71, 0x1d, 0xc4210fdde0, 0x1, 0x1)
        /Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/github.com/sirupsen/logrus/logger.go:181 +0x85
github.com/argoproj/argo-events/vendor/github.com/sirupsen/logrus.Panicf(0x157ff71, 0x1d, 0xc4210fdde0, 0x1, 0x1)
        /Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/github.com/sirupsen/logrus/exported.go:155 +0x5f
github.com/argoproj/argo-events/controller.(*SensorController).processNextItem(0xc420122820, 0xc4201c3200)
        /Users/mmagaldi/go/src/github.com/argoproj/argo-events/controller/controller.go:125 +0x3a4
github.com/argoproj/argo-events/controller.(*SensorController).runWorker(0xc420122820)
        /Users/mmagaldi/go/src/github.com/argoproj/argo-events/controller/controller.go:182 +0x2b
github.com/argoproj/argo-events/controller.(*SensorController).(github.com/argoproj/argo-events/controller.runWorker)-fm()
        /Users/mmagaldi/go/src/github.com/argoproj/argo-events/controller/controller.go:175 +0x2a
github.com/argoproj/argo-events/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc4208dc9b0)
        /Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
github.com/argoproj/argo-events/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc4208dc9b0, 0x3b9aca00, 0x0, 0x1, 0x0)
        /Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd
github.com/argoproj/argo-events/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc4208dc9b0, 0x3b9aca00, 0x0)
        /Users/mmagaldi/go/src/github.com/argoproj/argo-events/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by github.com/argoproj/argo-events/controller.(*SensorController).Run
        /Users/mmagaldi/go/src/github.com/argoproj/argo-events/controller/controller.go:175 +0x2f8

Expected behavior
It should not sendMessage if user not config Spec.Escalation.

@magaldima magaldima self-assigned this Aug 16, 2018
@magaldima
Copy link
Contributor

@fatkun thanks for the issue. this looks like an easy fix to just check if the configured sensor has a non nil escalation policy. I'll push up a branch with a fix!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants