Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Intermittent Cloud Event failures during integration tests #3374

Closed
pritidesai opened this issue Oct 12, 2020 · 2 comments · Fixed by #3375
Closed

Intermittent Cloud Event failures during integration tests #3374

pritidesai opened this issue Oct 12, 2020 · 2 comments · Fixed by #3375
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@pritidesai
Copy link
Member

pritidesai commented Oct 12, 2020 •
edited
Loading

I have noticed this kind of failure happening in many PRs recently.

/cc @afrittoli

TestTaskRunPipelineRunCancel/retries=0: cancel_test.go:159: Expected 2 number of successful events from pipelinerun and taskrun but got 3; list of received events : "&v1.Event{TypeMeta:v1.TypeMeta{Kind:\"\", APIVersion:\"\"}, ObjectMeta:v1.ObjectMeta{Name:\"cancel-me.163d53abc302932f\", GenerateName:\"\", Namespace:\"arendelle-khkkw\", SelfLink:\"/api/v1/namespaces/arendelle-khkkw/events/cancel-me.163d53abc302932f\", UID:\"cd2d7148-1b0d-41a4-b2f7-66d4575c78e0\", ResourceVersion:\"423\", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63738125840, loc:(*time.Location)(0x4215be0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:\"\", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:\"PipelineRun\", Namespace:\"arendelle-khkkw\", Name:\"cancel-me\", UID:\"766809d5-cc12-4e43-a4dd-2fb0524b4974\", APIVersion:\"tekton.dev/v1beta1\", ResourceVersion:\"2973\", FieldPath:\"\"}, Reason:\"Failed\", Message:\"PipelineRun \\\"cancel-me\\\" was cancelled\", Source:v1.EventSource{Component:\"PipelineRun\", Host:\"\"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63738125839, loc:(*time.Location)(0x4215be0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63738125839, loc:(*time.Location)(0x4215be0)}}, Count:1, Type:\"Warning\", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:\"\", Related:(*v1.ObjectReference)(nil), ReportingController:\"\", ReportingInstance:\"\"}, &v1.Event{TypeMeta:v1.TypeMeta{Kind:\"\", APIVersion:\"\"}, ObjectMeta:v1.ObjectMeta{Name:\"cancel-me-task-8dk65.163d53abde283563\", GenerateName:\"\", Namespace:\"arendelle-khkkw\", SelfLink:\"/api/v1/namespaces/arendelle-khkkw/events/cancel-me-task-8dk65.163d53abde283563\", UID:\"4f461f1a-61f7-4329-81e8-4eb519d10e5a\", ResourceVersion:\"428\", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63738125840, loc:(*time.Location)(0x4215be0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:\"\", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:\"TaskRun\", Namespace:\"arendelle-khkkw\", Name:\"cancel-me-task-8dk65\", UID:\"c2245e73-bde6-4ed5-9b31-f0e679a205c5\", APIVersion:\"tekton.dev/v1beta1\", ResourceVersion:\"2977\", FieldPath:\"\"}, Reason:\"Failed\", Message:\"TaskRun \\\"cancel-me-task-8dk65\\\" was cancelled\", Source:v1.EventSource{Component:\"TaskRun\", Host:\"\"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63738125840, loc:(*time.Location)(0x4215be0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63738125840, loc:(*time.Location)(0x4215be0)}}, Count:1, Type:\"Warning\", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:\"\", Related:(*v1.ObjectReference)(nil), ReportingController:\"\", ReportingInstance:\"\"}, &v1.Event{TypeMeta:v1.TypeMeta{Kind:\"\", APIVersion:\"\"}, ObjectMeta:v1.ObjectMeta{Name:\"cancel-me-task-8dk65.163d53abde283563\", GenerateName:\"\", Namespace:\"arendelle-khkkw\", SelfLink:\"/api/v1/namespaces/arendelle-khkkw/events/cancel-me-task-8dk65.163d53abde283563\", UID:\"4f461f1a-61f7-4329-81e8-4eb519d10e5a\", ResourceVersion:\"431\", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63738125840, loc:(*time.Location)(0x4215be0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:\"\", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:\"TaskRun\", Namespace:\"arendelle-khkkw\", Name:\"cancel-me-
task-8dk65\", UID:\"c2245e73-bde6-4ed5-9b31-f0e679a205c5\", APIVersion:\"tekton.dev/v1beta1\", ResourceVersion:\"2977\", FieldPath:\"\"}, Reason:\"Failed\", Message:\"TaskRun \\\"cancel-me-task-8dk65\\\" was cancelled\", Source:v1.EventSource{Component:\"TaskRun\", Host:\"\"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63738125840, loc:(*time.Location)(0x4215be0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63738125840, loc:(*time.Location)(0x4215be0)}}, Count:2, Type:\"Warning\", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:\"\", Related:(*v1.ObjectReference)(nil), ReportingController:\"\", ReportingInstance:\"\"}"
    TestTaskRunPipelineRunCancel/retries=0: init_test.go:74: ############################################
    TestTaskRunPipelineRunCancel/retries=0: init_test.go:75: ### Dumping objects from arendelle-khkkw ###
    TestTaskRunPipelineRunCancel/retries=0: init_test.go:76: ############################################
    TestTaskRunPipelineRunCancel/retries=0: panic.go:634: 
        ---
        apiVersion: tekton.dev/v1beta1
        kind: PipelineRun
        metadata:
          creationTimestamp: "2020-10-12T18:57:16Z"
          generation: 2
          labels:
            tekton.dev/pipeline: cancel-me
          name: cancel-me
          namespace: arendelle-khkkw
          resourceVersion: "3015"
          selfLink: /apis/tekton.dev/v1beta1/namespaces/arendelle-khkkw/pipelineruns/cancel-me
          uid: 766809d5-cc12-4e43-a4dd-2fb0524b4974
        spec:
          pipelineSpec:
            tasks:
            - name: task
              retries: 1
              taskSpec:
                metadata: {}
                steps:
                - image: busybox
                  name: ""
                  resources: {}
                  script: sleep 5000
          serviceAccountName: default
          status: PipelineRunCancelled
          timeout: 1h0m0s
        status:
          completionTime: "2020-10-12T18:57:19Z"
          conditions:
          - lastTransitionTime: "2020-10-12T18:57:19Z"
            message: PipelineRun "cancel-me" was cancelled
            reason: PipelineRunCancelled
            status: "False"
            type: Succeeded
          pipelineSpec:
            tasks:
            - name: task
              retries: 1
              taskSpec:
                metadata: {}
                steps:
                - image: busybox
                  name: ""
                  resources: {}
                  script: sleep 5000
          startTime: "2020-10-12T18:57:16Z"
          taskRuns:
            cancel-me-task-8dk65:
              pipelineTaskName: task
              status:
                completionTime: "2020-10-12T18:57:20Z"
                conditions:
                - lastTransitionTime: "2020-10-12T18:57:20Z"
                  message: TaskRun "cancel-me-task-8dk65" was cancelled
                  reason: TaskRunCancelled
                  status: "False"
                  type: Succeeded
                podName: cancel-me-task-8dk65-pod-v8cxb
                startTime: "2020-10-12T18:57:16Z"
                steps:
                - container: step-unnamed-0
                  name: unnamed-0
                  terminated:
                    exitCode: 1
                    finishedAt: "2020-10-12T18:57:20Z"
                    reason: TaskRunCancelled
                    startedAt: null
                taskSpec:
                  steps:
                  - image: busybox
                    name: ""
                    resources: {}
                    script: sleep 5000
        
        ---
        apiVersion: tekton.dev/v1beta1
        kind: TaskRun
        metadata:
          annotations:
            pipeline.tekton.dev/release: devel
          creationTimestamp: "2020-10-12T18:57:16Z"
          generation: 2
          labels:
            app.kubernetes.io/managed-by: tekton-pipelines
            tekton.dev/pipeline: cancel-me
            tekton.dev/pipelineRun: cancel-me
            tekton.dev/pipelineTask: task
          name: cancel-me-task-8dk65
          namespace: arendelle-khkkw
          ownerReferences:
          - apiVersion: tekton.dev/v1beta1
            blockOwnerDeletion: true
            controller: true
            kind: PipelineRun
            name: cancel-me
            uid: 766809d5-cc12-4e43-a4dd-2fb0524b4974
          resourceVersion: "3010"
          selfLink: /apis/tekton.dev/v1beta1/namespaces/arendelle-khkkw/taskruns/cancel-me-task-8dk65
          uid: c2245e73-bde6-4ed5-9b31-f0e679a205c5
        spec:
          resources: {}
          serviceAccountName: default
          status: TaskRunCancelled
          taskSpec:
            steps:
            - image: busybox
              name: ""
              resources: {}
              script: sleep 5000
          timeout: 1h0m0s
        status:
          completionTime: "2020-10-12T18:57:20Z"
          conditions:
          - lastTransitionTime: "2020-10-12T18:57:20Z"
            message: TaskRun "cancel-me-task-8dk65" was cancelled
            reason: TaskRunCancelled
            status: "False"
            type: Succeeded
          podName: cancel-me-task-8dk65-pod-v8cxb
          startTime: "2020-10-12T18:57:16Z"
          steps:
          - container: step-unnamed-0
            name: unnamed-0
            terminated:
              exitCode: 1
              finishedAt: "2020-10-12T18:57:20Z"
              reason: TaskRunCancelled
              startedAt: null
          taskSpec:
            steps:
            - image: busybox
              name: ""
              resources: {}
              script: sleep 5000
        
    TestTaskRunPipelineRunCancel/retries=0: init_test.go:74: #####################################################
    TestTaskRunPipelineRunCancel/retries=0: init_test.go:75: ### Dumping logs from Pods in the arendelle-khkkw ###
    TestTaskRunPipelineRunCancel/retries=0: init_test.go:76: #####################################################
    TestTaskRunPipelineRunCancel/retries=0: build_logs.go:34: Could not get logs for pod cancel-me-task-8dk65-pod-v8cxb: pods "cancel-me-task-8dk65-pod-v8cxb" not found
@pritidesai pritidesai added the kind/bug Categorizes issue or PR as related to a bug. label Oct 12, 2020
@afrittoli
Copy link
Member

Thanks for the report!
The issues from the logs seems to be about k8s events, not cloud events.
This looks like a duplicate of #3366, so I'll close this one.

@afrittoli afrittoli mentioned this issue Oct 12, 2020
Closed
afrittoli added a commit to afrittoli/pipeline that referenced this issue Oct 12, 2020
@afrittoli
WIP Check that events are at least expected ones
c0242f5 
Fixes tektoncd#3374

Signed-off-by: Andrea Frittoli <[email protected]>
@afrittoli afrittoli mentioned this issue Oct 12, 2020
Merged
4 tasks
@pritidesai
Copy link
Member Author

pritidesai commented Oct 12, 2020 •
edited
Loading

no wonder this was not fixed with 3353 😱 🤦‍♀️

afrittoli added a commit to afrittoli/pipeline that referenced this issue Oct 13, 2020
@afrittoli
Check that events are at least expected ones
43d7ad1 
Events are not guaranteed to be sent once, so allow for more than
one in the cancel_test.

Fixes tektoncd#3374

Signed-off-by: Andrea Frittoli <[email protected]>
afrittoli added a commit to afrittoli/pipeline that referenced this issue Oct 13, 2020
@afrittoli
Check that events are at least expected ones
7a3b445 
Events are not guaranteed to be sent once, so allow for more than
one in the cancel_test.

Fixes tektoncd#3374

Signed-off-by: Andrea Frittoli <[email protected]>
afrittoli added a commit to afrittoli/pipeline that referenced this issue Oct 15, 2020
@afrittoli
Check that events are at least expected ones
037ddf7 
Events are not guaranteed to be sent once, so allow for more than
one in the cancel_test.

Fixes tektoncd#3374

Signed-off-by: Andrea Frittoli <[email protected]>
afrittoli added a commit to afrittoli/pipeline that referenced this issue Oct 20, 2020
@afrittoli
Check that events are at least expected ones
e87d05b 
Events are not guaranteed to be sent once, so allow for more than
one in the cancel_test.

Fixes tektoncd#3374

Signed-off-by: Andrea Frittoli <[email protected]>
afrittoli added a commit to afrittoli/pipeline that referenced this issue Oct 20, 2020
@afrittoli
Check that events are at least expected ones
53c2be5 
Events are not guaranteed to be sent once, so allow for more than
one in the cancel_test.

Fixes tektoncd#3374

Signed-off-by: Andrea Frittoli <[email protected]>
tekton-robot pushed a commit that referenced this issue Oct 20, 2020
@afrittoli @tekton-robot
Check that events are at least expected ones
e98baae 
Events are not guaranteed to be sent once, so allow for more than
one in the cancel_test.

Fixes #3374

Signed-off-by: Andrea Frittoli <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Check that events are at least expected ones afrittoli/pipeline
2 participants
@pritidesai @afrittoli