Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resource Event Reprocesses 'Events' and Doesn't Support Annotation Filters #560

Closed
bpayt2 opened this issue Mar 22, 2020 · 18 comments · Fixed by #571
Closed

Resource Event Reprocesses 'Events' and Doesn't Support Annotation Filters #560

bpayt2 opened this issue Mar 22, 2020 · 18 comments · Fixed by #571

Comments

@bpayt2
Copy link

bpayt2 commented Mar 22, 2020

The resource event source type is performing a list operation for a particular resources within Kubernetes. This is not an 'event/trigger' but just a continual list. I am reprocessing the same resources continuously. Can you add support for only new resources?

Also, I am unable to filter on annotations.

@VaibhavPage
Copy link
Contributor

VaibhavPage commented Mar 22, 2020

Are you using v0.13.0? Take a look at the updated event-source for resource gateway at https://github.com/argoproj/argo-events/blob/master/examples/event-sources/resource.yaml . The type is renamed to eventType.

The resource gateway runs a shared informer for events (Add, Update and Delete) and doesn't just list objects in a for-loop. If you want to listen to new objects, then you need to specify the correct eventType(ADD, UPDATE or DELETE). You can add filters on labels to narrow the list of objects you want to monitor.

To listen to new objects. specify eventType as ADD.

The gateway does not process any object twice. It only processes objects notified to it by Kubernetes API server.

Annotations are not meant to be used for filtering purposes. They contain non-identifying information as specified in Kubernetes guidelines. Besides K8s Informer API does not allow to filter on annotations. If you still want to filter on annotations, I'd suggest using DataFilters in Sensor https://argoproj.github.io/argo-events/tutorials/07-filters/#data-filter.

There is a pr #561 to support a list of event types for resource gateway which will be available in next release.

@bpayt2
Copy link
Author

bpayt2 commented Mar 23, 2020

I am using eventType and Label filters; however, I am not getting the expected behavior.

Expected Behavior: Generate event each time a new AWS EBS persistentvolume is created.
Actual Behavior: A list of all EBS volumes is pulled when the event source is created. I am seeing duplicate processes. The label filter works; but the createdBy is historic and doesn't allow me to specify 'from this point on'. The request for Annotation filter is because I want to use system information within my selection; example: 'from this storage class provisioner'.

Data filters are not working with the Sensor because Kubernetes resource event data is base64 encoded - once decoded, the body is also encoded. I do not see where I can specify dataDecoded (for example): as a data filter within the sensor. This is preventing me from mapping event data to parameters as well.

############Event Source File###############
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
name: resource-event-source
spec:
type: "resource"
resource:
example:
namespace: ""
group: ""
version: "v1"
resource: "persistentvolumes"
type: ADD
filter:
labels:
failure-domain.beta.kubernetes.io/region: "us-east-1"

@bpayt2
Copy link
Author

bpayt2 commented Mar 23, 2020

The reprocessing that is occurring also do to the base64 encoding issue. The eventType is within the data element that is encoded; so it is not filtering correctly. I am getting all updates, add, deletes.

@bpayt2
Copy link
Author

bpayt2 commented Mar 26, 2020

update?

@VaibhavPage
Copy link
Contributor

VaibhavPage commented Mar 28, 2020

Everything seems to work. Let's take an example,

  1. The resource gateway set up is available here

  2. The structure of the event is as follows,

    {
        "context": {
          "type": "type_of_gateway",
          "specVersion": "cloud_events_version",
          "source": "name_of_the_gateway",
          "eventID": "unique_event_id",
          "time": "event_time",
          "dataContentType": "type_of_data",
          "subject": "name_of_the_event_within_event_source"
        },
        "data": {
          "type": "type_of_the_event", // ADD, UPDATE or DELETE
          "body": "resource_body", // JSON format
          "group": "resource_group_name",
          "version": "resource_version_name",
          "resource": "resource_name"
        }
    }
  1. Let set up gateway to listen to CREATE events for workflow with label name: my-worfklow,
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: resource-event-source
spec:
  type: "resource"
  resource:
    example:
      # namespace to listen events within
      namespace: "argo-events"
      # resource group
      group: "argoproj.io"
      # resource version
      version: "v1alpha1"
      # resource kind
      resource: "workflows"
      # type of event
      # possible values are ADD, DELETE, UPDATE
      # Optional
      eventType: ADD
      # Filters to apply on watched object
      # Optional
      filter:
        labels:
          name: "my-workflow"

4.Set up the gateway

apiVersion: argoproj.io/v1alpha1
kind: Gateway
metadata:
  name: resource-gateway
  labels:
    # gateway controller with instanceId "argo-events" will process this gateway
    gateways.argoproj.io/gateway-controller-instanceid: argo-events
spec:
  type: resource
  eventSourceRef:
    name: resource-event-source
  template:
    metadata:
      name: resource-gateway
      labels:
        gateway-name: resource-gateway
    spec:
      containers:
        - name: gateway-client
          image: argoproj/gateway-client:v0.13.0
          imagePullPolicy: Always
          command: ["/bin/gateway-client"]
        - name: resource-events
          image: argoproj/resource-gateway:v0.13.0
          imagePullPolicy: Always
          command: ["/bin/resource-gateway"]
      serviceAccountName: argo-events-sa
  subscribers:
    http:
      - "http://resource-sensor.argo-events.svc:9300/"

  1. Create the sensor. Let's parameterize the triggered workflow with the name of the workflow which caused the event. I've added the data filter as well.

apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: resource-sensor
  labels:
    sensors.argoproj.io/sensor-controller-instanceid: argo-events
spec:
  template:
    spec:
      containers:
        - name: sensor
          image: argoproj/sensor:v0.13.0
          imagePullPolicy: Always
      serviceAccountName: argo-events-sa
  subscription:
    http:
      port: 9300
  dependencies:
    - name: test-dep
      gatewayName: resource-gateway
      eventName: example
      filters:
        name: data-filter
        data:
          - path: body.metadata.name
            type: string
            value:
              - my-workflow
  triggers:
    - template:
        name: argo-workflow
        k8s:
          group: argoproj.io
          version: v1alpha1
          resource: workflows
          operation: create
          source:
            resource:
              apiVersion: argoproj.io/v1alpha1
              kind: Workflow
              metadata:
                generateName: resource-workflow-
              spec:
                entrypoint: whalesay
                arguments:
                  parameters:
                  - name: message
                    # the value will get overridden by the event payload from test-dep
                    value: hello world
                templates:
                - name: whalesay
                  inputs:
                    parameters:
                    - name: message
                  container:
                    image: docker/whalesay:latest
                    imagePullPolicy: IfNotPresent
                    command: [cowsay]
                    args: ["{{inputs.parameters.message}}"]
          parameters:
            - src:
                dependencyName: test-dep
                dataKey: body.metadata.name
              dest: spec.arguments.parameters.0.value

  1. Create the workflow my-workflow,
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: my-workflow
  labels:
    name: my-workflow
spec:
  entrypoint: whalesay
  templates:
  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [cowsay]
      args: ["hello world"]
  1. Take a look at the trigger workflow resource-workflow-xxxxxx, it prints,

  | _____________
-- | --
  | < my-workflow >
  | -------------
  | \
  | \
  | \
  | ##        .
  | ## ## ##       ==
  | ## ## ## ##      ===
  | /""""""""""""""""___/ ===
  | ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
  | \______ o          __/
  | \    \        __/
  | \____\______/

  1. Change the data filters however you like to filter on annotations and other things...

  2. You can find more info on trigger parameterization at https://argoproj.github.io/argo-events/tutorials/02-parameterization/ and data filters at https://argoproj.github.io/argo-events/tutorials/07-filters/

@gordiustechnologies
Copy link

gordiustechnologies commented Mar 29, 2020

@VaibhavPage, I have the same problem. I am trying to react to node deletions using resource events and the eventType mechanism doesn't seem to be working. Not just deletions, but all node events seem to go through. Generated events look like:

{ 'context': { 'dataContentType': 'application/json',
               'id': '32373863653939352d333766332d343664612d386665322d623766363532643363393462',
               'source': 'resource-gateway',
               'specVersion': '0.3',
               'subject': 'node-delete',
               'time': '2020-03-29T02:58:57.420038Z',
               'type': 'resource'},
  'data': 'eyJ0eXBlIjoiVVBEQVRFIiwiYm9keSI6eyJhcGlWZXJzaW9uIjoidjEiLCJraW5kIjoiTm9kZSIsIm1ldGFkYXRhIjp7ImFubm90YXRpb25zIjp7Imt1YmVhZG0uYWxwaGEua3ViZXJuZXRlcy5pby9jcmktc29ja2V0IjoiL3Zhci9ydW4vZG9ja2Vyc2hpbS5zb2NrIiwibm9kZS5hbHBoYS5rdWJlcm5ldGVzLmlvL3R0bCI6IjAiLCJwcm9qZWN0Y2FsaWNvLm9yZy9JUHY0QWRkcmVzcyI6IjE3Mi4zMi4yNDAuMTYwLzE4IiwicHJvamVjdGNhbGljby5vcmcvSVB2NElQSVBUdW5uZWxBZGRyIjoiMTkyLjE2OC41My4wIiwidm9sdW1lcy5rdWJlcm5ldGVzLmlvL2NvbnRyb2xsZXItbWFuYWdlZC1hdHRhY2gtZGV0YWNoIjoidHJ1ZSJ9LCJjcmVhdGlvblRpbWVzdGFtcCI6IjIwMjAtMDMtMjlUMDI6NTY6NTNaIiwibGFiZWxzIjp7ImJldGEua3ViZXJuZXRlcy5pby9hcmNoIjoiYW1kNjQiLCJiZXRhLmt1YmVybmV0ZXMuaW8vaW5zdGFuY2UtdHlwZSI6InQzLnhsYXJnZSIsImJldGEua3ViZXJuZXRlcy5pby9vcyI6ImxpbnV4IiwiZmFpbHVyZS1kb21haW4uYmV0YS5rdWJlcm5ldGVzLmlvL3JlZ2lvbiI6InVzLXdlc3QtMiIsImZhaWx1cmUtZG9tYWluLmJldGEua3ViZXJuZXRlcy5pby96b25lIjoidXMtd2VzdC0yZCIsImt1YmVybmV0ZXMuaW8vYXJjaCI6ImFtZDY0Iiwia3ViZXJuZXRlcy5pby9ob3N0bmFtZSI6ImlwLTE3Mi0zMi0yNDAtMTYwLnVzLXdlc3QtMi5jb21wdXRlLmludGVybmFsIiwia3ViZXJuZXRlcy5pby9vcyI6ImxpbnV4Iiwibm9kZS1yb2xlLmt1YmVybmV0ZXMuaW8vbWFzdGVyIjoiIn0sIm1hbmFnZWRGaWVsZHMiOlt7ImFwaVZlcnNpb24iOiJ2MSIsImZpZWxkc1R5cGUiOiJGaWVsZHNWMSIsImZpZWxkc1YxIjp7ImY6bWV0YWRhdGEiOnsiZjpsYWJlbHMiOnsiZjpiZXRhLmt1YmVybmV0ZXMuaW8vaW5zdGFuY2UtdHlwZSI6e30sImY6ZmFpbHVyZS1kb21haW4uYmV0YS5rdWJlcm5ldGVzLmlvL3JlZ2lvbiI6e30sImY6ZmFpbHVyZS1kb21haW4uYmV0YS5rdWJlcm5ldGVzLmlvL3pvbmUiOnt9fX0sImY6c3BlYyI6eyJmOnByb3ZpZGVySUQiOnt9LCJmOnRhaW50cyI6e319LCJmOnN0YXR1cyI6eyJmOmFkZHJlc3NlcyI6eyJrOntcInR5cGVcIjpcIkV4dGVybmFsRE5TXCJ9Ijp7Ii4iOnt9LCJmOmFkZHJlc3MiOnt9LCJmOnR5cGUiOnt9fSwiazp7XCJ0eXBlXCI6XCJFeHRlcm5hbElQXCJ9Ijp7Ii4iOnt9LCJmOmFkZHJlc3MiOnt9LCJmOnR5cGUiOnt9fSwiazp7XCJ0eXBlXCI6XCJJbnRlcm5hbEROU1wifSI6eyIuIjp7fSwiZjphZGRyZXNzIjp7fSwiZjp0eXBlIjp7fX19fX0sIm1hbmFnZXIiOiJjbG91ZC1jb250cm9sbGVyLW1hbmFnZXIiLCJvcGVyYXRpb24iOiJVcGRhdGUiLCJ0aW1lIjoiMjAyMC0wMy0yOVQwMjo1Nzo1MVoifSx7ImFwaVZlcnNpb24iOiJ2MSIsImZpZWxkc1R5cGUiOiJGaWVsZHNWMSIsImZpZWxkc1YxIjp7ImY6c3RhdHVzIjp7ImY6Y29uZGl0aW9ucyI6eyJrOntcInR5cGVcIjpcIkRpc2tQcmVzc3VyZVwifSI6eyJmOmxhc3RIZWFydGJlYXRUaW1lIjp7fX0sIms6e1widHlwZVwiOlwiTWVtb3J5UHJlc3N1cmVcIn0iOnsiZjpsYXN0SGVhcnRiZWF0VGltZSI6e319LCJrOntcInR5cGVcIjpcIlBJRFByZXNzdXJlXCJ9Ijp7ImY6bGFzdEhlYXJ0YmVhdFRpbWUiOnt9fSwiazp7XCJ0eXBlXCI6XCJSZWFkeVwifSI6eyJmOmxhc3RIZWFydGJlYXRUaW1lIjp7fX19LCJmOmltYWdlcyI6e319fSwibWFuYWdlciI6Imt1YmVsZXQiLCJvcGVyYXRpb24iOiJVcGRhdGUiLCJ0aW1lIjoiMjAyMC0wMy0yOVQwMjo1ODo1N1oifV0sIm5hbWUiOiJpcC0xNzItMzItMjQwLTE2MC51cy13ZXN0LTIuY29tcHV0ZS5pbnRlcm5hbCIsInJlc291cmNlVmVyc2lvbiI6IjExNDUiLCJzZWxmTGluayI6Ii9hcGkvdjEvbm9kZXMvaXAtMTcyLTMyLTI0MC0xNjAudXMtd2VzdC0yLmNvbXB1dGUuaW50ZXJuYWwiLCJ1aWQiOiJiZmM0MmRhZi0wNzJmLTQ0YTMtODRiNi1jMWFlZjMyMzRiMWMifSwic3BlYyI6eyJwb2RDSURSIjoiMTkyLjE2OC4wLjAvMjQiLCJwb2RDSURScyI6WyIxOTIuMTY4LjAuMC8yNCJdLCJwcm92aWRlcklEIjoiYXdzOi8vL3VzLXdlc3QtMmQvaS0wZTAyNTE1NGEyMzFkYWQ0ZCIsInRhaW50cyI6W3siZWZmZWN0IjoiTm9TY2hlZHVsZSIsImtleSI6Im5vZGUtcm9sZS5rdWJlcm5ldGVzLmlvL21hc3RlciJ9XX0sInN0YXR1cyI6eyJhZGRyZXNzZXMiOlt7ImFkZHJlc3MiOiIxNzIuMzIuMjQwLjE2MCIsInR5cGUiOiJJbnRlcm5hbElQIn0seyJhZGRyZXNzIjoiNDQuMjI5LjMxLjcwIiwidHlwZSI6IkV4dGVybmFsSVAifSx7ImFkZHJlc3MiOiJpcC0xNzItMzItMjQwLTE2MC51cy13ZXN0LTIuY29tcHV0ZS5pbnRlcm5hbCIsInR5cGUiOiJJbnRlcm5hbEROUyJ9LHsiYWRkcmVzcyI6ImlwLTE3Mi0zMi0yNDAtMTYwLnVzLXdlc3QtMi5jb21wdXRlLmludGVybmFsIiwidHlwZSI6Ikhvc3RuYW1lIn0seyJhZGRyZXNzIjoiZWMyLTQ0LTIyOS0zMS03MC51cy13ZXN0LTIuY29tcHV0ZS5hbWF6b25hd3MuY29tIiwidHlwZSI6IkV4dGVybmFsRE5TIn1dLCJhbGxvY2F0YWJsZSI6eyJjcHUiOiI0IiwiZXBoZW1lcmFsLXN0b3JhZ2UiOiI3NDMwMjg5Mzk2IiwiaHVnZXBhZ2VzLTFHaSI6IjAiLCJodWdlcGFnZXMtMk1pIjoiMCIsIm1lbW9yeSI6IjE1NzU5TWkiLCJwb2RzIjoiMTEwIn0sImNhcGFjaXR5Ijp7ImNwdSI6IjQiLCJlcGhlbWVyYWwtc3RvcmFnZSI6IjgwNjIzODBLaSIsImh1Z2VwYWdlcy0xR2kiOiIwIiwiaHVnZXBhZ2VzLTJNaSI6IjAiLCJtZW1vcnkiOiIxNTg1OU1pIiwicG9kcyI6IjExMCJ9LCJjb25kaXRpb25zIjpbeyJsYXN0SGVhcnRiZWF0VGltZSI6IjIwMjAtMDMtMjlUMDI6NTc6NDBaIiwibGFzdFRyYW5zaXRpb25UaW1lIjoiMjAyMC0wMy0yOVQwMjo1Nzo0MFoiLCJtZXNzYWdlIjoiQ2FsaWNvIGlzIHJ1bm5pbmcgb24gdGhpcyBub2RlIiwicmVhc29uIjoiQ2FsaWNvSXNVcCIsInN0YXR1cyI6IkZhbHNlIiwidHlwZSI6Ik5ldHdvcmtVbmF2YWlsYWJsZSJ9LHsibGFzdEhlYXJ0YmVhdFRpbWUiOiIyMDIwLTAzLTI5VDAyOjU4OjU3WiIsImxhc3RUcmFuc2l0aW9uVGltZSI6IjIwMjAtMDMtMjlUMDI6NTY6NTNaIiwibWVzc2FnZSI6Imt1YmVsZXQgaGFzIHN1ZmZpY2llbnQgbWVtb3J5IGF2YWlsYWJsZSIsInJlYXNvbiI6Ikt1YmVsZXRIYXNTdWZmaWNpZW50TWVtb3J5Iiwic3RhdHVzIjoiRmFsc2UiLCJ0eXBlIjoiTWVtb3J5UHJlc3N1cmUifSx7Imxhc3RIZWFydGJlYXRUaW1lIjoiMjAyMC0wMy0yOVQwMjo1ODo1N1oiLCJsYXN0VHJhbnNpdGlvblRpbWUiOiIyMDIwLTAzLTI5VDAyOjU2OjUzWiIsIm1lc3NhZ2UiOiJrdWJlbGV0IGhhcyBubyBkaXNrIHByZXNzdXJlIiwicmVhc29uIjoiS3ViZWxldEhhc05vRGlza1ByZXNzdXJlIiwic3RhdHVzIjoiRmFsc2UiLCJ0eXBlIjoiRGlza1ByZXNzdXJlIn0seyJsYXN0SGVhcnRiZWF0VGltZSI6IjIwMjAtMDMtMjlUMDI6NTg6NTdaIiwibGFzdFRyYW5zaXRpb25UaW1lIjoiMjAyMC0wMy0yOVQwMjo1Njo1M1oiLCJtZXNzYWdlIjoia3ViZWxldCBoYXMgc3VmZmljaWVudCBQSUQgYXZhaWxhYmxlIiwicmVhc29uIjoiS3ViZWxldEhhc1N1ZmZpY2llbnRQSUQiLCJzdGF0dXMiOiJGYWxzZSIsInR5cGUiOiJQSURQcmVzc3VyZSJ9LHsibGFzdEhlYXJ0YmVhdFRpbWUiOiIyMDIwLTAzLTI5VDAyOjU4OjU3WiIsImxhc3RUcmFuc2l0aW9uVGltZSI6IjIwMjAtMDMtMjlUMDI6NTc6MzdaIiwibWVzc2FnZSI6Imt1YmVsZXQgaXMgcG9zdGluZyByZWFkeSBzdGF0dXMuIEFwcEFybW9yIGVuYWJsZWQiLCJyZWFzb24iOiJLdWJlbGV0UmVhZHkiLCJzdGF0dXMiOiJUcnVlIiwidHlwZSI6IlJlYWR5In1dLCJkYWVtb25FbmRwb2ludHMiOnsia3ViZWxldEVuZHBvaW50Ijp7IlBvcnQiOjEwMjUwfX0sImltYWdlcyI6W3sibmFtZXMiOlsiYXJnb3Byb2ovc2Vuc29yQHNoYTI1NjozNWExOTkxOWE0OWRhMWRiMGEyMzhiZDFhNmVlZTMxNjM4MTRkNzYzYzRhM2E2MmRlMWU3M2Y3MTkxMzBmYTQ2IiwiYXJnb3Byb2ovc2Vuc29yOnYwLjEzLjAiXSwic2l6ZUJ5dGVzIjo0ODMxNDg2MzR9LHsibmFtZXMiOlsiYXJnb3Byb2ovYXJnb2V4ZWNAc2hhMjU2OjNiYmVjM2ViN2MwMWQ2NzQ3ZTliZmNkOGI2ZjU4M2NhMzc5Y2IwZWZjYThiYmUyMjZhYTIyODYzYTk5MGJmMWQiLCJhcmdvcHJvai9hcmdvZXhlYzp2Mi43LjAtcmMzIl0sInNpemVCeXRlcyI6Mjk4NDgyNjU4fSx7Im5hbWVzIjpbImdvcmRpdXMvazhzLWF3c0BzaGEyNTY6YjhiNTZlOWU1NGEyYzJjMzZjODRiODRhNGE4YWQ4N2UxMzViMmU0Y2QyOTUwYzc4MGM0MzI1OTliMGFkMTdjYyIsImdvcmRpdXMvazhzLWF3czpsYXRlc3QiXSwic2l6ZUJ5dGVzIjoyOTQ4NDE3Mzl9LHsibmFtZXMiOlsiazhzLmdjci5pby9ldGNkQHNoYTI1Njo0YWZiOTliNDY5MGI0MThmZmMyY2ViNjdlMWExNzM3NjQ1N2U0NDFjMWYwOWFiNTU0NDdmMGFhZjk5MmZhNjQ2IiwiazhzLmdjci5pby9ldGNkOjMuNC4zLTAiXSwic2l6ZUJ5dGVzIjoyODg0MjY5MTd9LHsibmFtZXMiOlsiY2FsaWNvL25vZGVAc2hhMjU2Ojg4N2JjZDU1MTY2OGNjY2FlMWZiZmQ2ZDJlYjBmNjM1ZWMzN2JiNGNmNTk5ZTExNjk5ODlhYTQ5ZGZhYzViNTciLCJjYWxpY28vbm9kZTp2My4xMS4yIl0sInNpemVCeXRlcyI6MjU1MzQzOTYyfSx7Im5hbWVzIjpbImFyZ29wcm9qL2dhdGV3YXktY2xpZW50QHNoYTI1Njo4YjZjNzFkMDg1NTY0ZjYxNjNhZjVhOTkyMzU5MTJmNWJhNjdlNGI1OTk4ZTk1ZTUzMDQ4MTA3NmUyOTcwZDA1IiwiYXJnb3Byb2ovZ2F0ZXdheS1jbGllbnQ6djAuMTMuMCJdLCJzaXplQnl0ZXMiOjI0ODIzOTg3OH0seyJuYW1lcyI6WyJhcmdvcHJvai9zZW5zb3ItY29udHJvbGxlckBzaGEyNTY6ZmRlMzE2NmM2NTU3ZTJlZjEyZWM3NDk0YzJmOTk2MmM0Mjg1NWM3YjlkMDdkMzNjOWE2NzdhMmE1ODMyNDc0ZiIsImFyZ29wcm9qL3NlbnNvci1jb250cm9sbGVyOnYwLjEzLjAiXSwic2l6ZUJ5dGVzIjoyNDM2NzEzODV9LHsibmFtZXMiOlsiYXJnb3Byb2ovZ2F0ZXdheS1jb250cm9sbGVyQHNoYTI1Njo1OWM1OWI2N2E4ZWEzYjgwZWIxNDIzMDk1ZDg5ODAwMzdkMWQ2OGY5YjQ5ZTlkOTFhZDY5MWE3ZTAxOGExMDY4IiwiYXJnb3Byb2ovZ2F0ZXdheS1jb250cm9sbGVyOnYwLjEzLjAiXSwic2l6ZUJ5dGVzIjoyNDMyNjkxNTl9LHsibmFtZXMiOlsiY2FsaWNvL2NuaUBzaGEyNTY6ZjU4MDg0MDFhOTZiYTkzMDEwYjk2OTMwMTk0OTZkODgwNzBkZGU4MGRkYTY5NzZkMTBiYzQzMjhmMWYxOGY0ZSIsImNhbGljby9jbmk6djMuMTEuMiJdLCJzaXplQnl0ZXMiOjIwNDE4NTc1M30seyJuYW1lcyI6WyJrOHMuZ2NyLmlvL2t1YmUtYXBpc2VydmVyQHNoYTI1NjpmYzRlZmI1NWMyYTdkNGU3YjlhODU4YzY3ZTI0ZjAwZTczOWRmNGVmNTA4MjUwMGMyYjYwZWEwOTAzZjE4MjQ4IiwiazhzLmdjci5pby9rdWJlLWFwaXNlcnZlcjp2MS4xOC4wIl0sInNpemVCeXRlcyI6MTcyOTY0MzcxfSx7Im5hbWVzIjpbIms4cy5nY3IuaW8va3ViZS1jb250cm9sbGVyLW1hbmFnZXJAc2hhMjU2OmQ5MjZiMTcyYjhmYzM1NjhkOGVhYjk3MzZiMmY3OWExYWZlZmNiODA5ZDAzMGRkNDQ2NWNiYjBkNDQ0Y2UyOTMiLCJrOHMuZ2NyLmlvL2t1YmUtY29udHJvbGxlci1tYW5hZ2VyOnYxLjE4LjAiXSwic2l6ZUJ5dGVzIjoxNjIzNjgwMTl9LHsibmFtZXMiOlsiazhzLmdjci5pby9jbG91ZC1jb250cm9sbGVyLW1hbmFnZXJAc2hhMjU2Ojg1OTgyNmMzNGUxNTUxZTcxNTFhYmM4NmZiMzI4NmVkZGMyZWI1M2U2OTdjODU5NGUyZTM5ZjEzNjZhNjE2ZTMiLCJrOHMuZ2NyLmlvL2Nsb3VkLWNvbnRyb2xsZXItbWFuYWdlcjp2MS4xNS4xMSJdLCJzaXplQnl0ZXMiOjE0MzAxMTU2OX0seyJuYW1lcyI6WyJrOHMuZ2NyLmlvL2t1YmUtcHJveHlAc2hhMjU2OjllODU4Mzg2ZDUyZDBhYmFmOTM2YzFkMTBhNzYzNjQ4YWI3ZDg1YzhlYjBhZjA4YTUwYTY0MjM4MTQ2ZTU1NzEiLCJrOHMuZ2NyLmlvL2t1YmUtcHJveHk6djEuMTguMCJdLCJzaXplQnl0ZXMiOjExNjUzNDI2M30seyJuYW1lcyI6WyJjYWxpY28vcG9kMmRhZW1vbi1mbGV4dm9sQHNoYTI1Njo5M2M2NGQ2ZTNlMGEwZGM3NWQxYjIxOTc0ZGIwNWQyOGVmMjE2MmJkOTE2YjAwY2U2MmEzOWZkMjM1OTRmODEwIiwiY2FsaWNvL3BvZDJkYWVtb24tZmxleHZvbDp2My4xMS4yIl0sInNpemVCeXRlcyI6MTExMTIyMzI0fSx7Im5hbWVzIjpbIms4cy5nY3IuaW8va3ViZS1zY2hlZHVsZXJAc2hhMjU2OjMzMDYzYmM4NTZlOTlkMTJiOWNiMzBhYWIxYzFjNzU1ZWNkNDU4ZDViZDEzMDI3MGRhN2M1MWM3MGNhMTBjZjYiLCJrOHMuZ2NyLmlvL2t1YmUtc2NoZWR1bGVyOnYxLjE4LjAiXSwic2l6ZUJ5dGVzIjo5NTI3NTUzOX0seyJuYW1lcyI6WyJhcmdvcHJvai9hcmdvY2xpQHNoYTI1Njo5MWJhNjU3YzkyM2ZjOWJkNWExZTVhNGZlZmZmMTJhNTZhZjI0MWQyM2UyYmIzMzQ2MWQ5MjdlYjYxMzJmMTBhIiwiYXJnb3Byb2ovYXJnb2NsaTp2Mi43LjAtcmMzIl0sInNpemVCeXRlcyI6Njg0MTczMDl9LHsibmFtZXMiOlsiY2FsaWNvL2t1YmUtY29udHJvbGxlcnNAc2hhMjU2OjExNjljY2E0MGI0ODkyNzE3MTRjYjFlOTdmZWQ5YjZiMTk4YWFiZGNhMWExY2M2MTY5OGRkNzNlZTY3MDNkNjAiLCJjYWxpY28va3ViZS1jb250cm9sbGVyczp2My4xMS4yIl0sInNpemVCeXRlcyI6NTI0Nzc5ODB9LHsibmFtZXMiOlsiYXJnb3Byb2ovd29ya2Zsb3ctY29udHJvbGxlckBzaGEyNTY6YzZlZmQ0OTQxYWFmMTEyODFkYzRiNDA4MzliMTllOTAyNTczY2MyMjljZmNkZWZiNjcwNThiMzRmYTkwYmU2ZiIsImFyZ29wcm9qL3dvcmtmbG93LWNvbnRyb2xsZXI6djIuNy4wLXJjMyJdLCJzaXplQnl0ZXMiOjUwMzMyMDQwfSx7Im5hbWVzIjpbImFyZ29wcm9qL3Jlc291cmNlLWdhdGV3YXlAc2hhMjU2OjNiMDI3NmEyNTVlYzI1ZTlhODBhOWM4YmYxNmFiZDhmNzgyOTJmYmNkNzA2ZmVjYjA0M2JhNDc2YjdhYmZhYTAiLCJhcmdvcHJvai9yZXNvdXJjZS1nYXRld2F5OnYwLjEzLjAiXSwic2l6ZUJ5dGVzIjo0NTU1NTE0NH0seyJuYW1lcyI6WyJrOHMuZ2NyLmlvL2NvcmVkbnNAc2hhMjU2OjJjOGQ2MWM0NmY0ODRkODgxZGI0M2IzNGQxM2NhNDdhMjY5MzM2ZTU3NmM4MWNmMDA3Y2E3NDBmYTllYzA4MDAiLCJrOHMuZ2NyLmlvL2NvcmVkbnM6MS42LjciXSwic2l6ZUJ5dGVzIjo0Mzc5NDE0N30seyJuYW1lcyI6WyJrOHMuZ2NyLmlvL3BhdXNlQHNoYTI1Njo5MjdkOTgxOTdlYzExNDFhMzY4NTUwODIyZDE4ZmExYzYwYmRhZTI3Yjc4YjBjMDA0ZjcwNWY1NDhjMDc4MTRmIiwiazhzLmdjci5pby9wYXVzZTozLjIiXSwic2l6ZUJ5dGVzIjo2ODI2OTZ9XSwibm9kZUluZm8iOnsiYXJjaGl0ZWN0dXJlIjoiYW1kNjQiLCJib290SUQiOiJkNWE5OWM1ZC04MTM1LTQ4MWQtOTc3My1mMTYxOWJmNTY2MmYiLCJjb250YWluZXJSdW50aW1lVmVyc2lvbiI6ImRvY2tlcjovLzE5LjMuOCIsImtlcm5lbFZlcnNpb24iOiI0LjE5LjAtOC1jbG91ZC1hbWQ2NCIsImt1YmVQcm94eVZlcnNpb24iOiJ2MS4xOC4wIiwia3ViZWxldFZlcnNpb24iOiJ2MS4xOC4wIiwibWFjaGluZUlEIjoiYTAzZjc3ODk1MmRkNDkzMTg5ZmYzZDgwMThjZTJiNWQiLCJvcGVyYXRpbmdTeXN0ZW0iOiJsaW51eCIsIm9zSW1hZ2UiOiJEZWJpYW4gR05VL0xpbnV4IDEwIChidXN0ZXIpIiwic3lzdGVtVVVJRCI6ImVjMjk1N2I2LWI4NTYtNmI3NC1hOTU1LTVlY2ViNzdhN2U5MSJ9fX0sImdyb3VwIjoiIiwidmVyc2lvbiI6InYxIiwicmVzb3VyY2UiOiJub2RlcyJ9'}

Note that this does not look like the example you gave: The data entry is not a dictionary, it is an encoded string (just like @bpayt2 observed). Could this be the cause of the issue?
For completeness, here is my event source:

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: resource-event-source
spec:
  type: resource
  resource:
    node-delete:
      group: ""
      version: v1
      resource: nodes
      eventType: DELETE

@ozankabak
Copy link

This is a continuation of the post above (coming from the same team). For now, we use a data filter in the sensor as a workaround to weed out non-delete events. The data filter looks like this:

...
dependencies:
  - name: resource-gateway-node-delete
    gatewayName: resource-gateway
    eventName: node-delete
    # # # WORKAROUND # # #
    filters:
      name: type-filter
      data:
        - path: type
          type: string
          value:
            - DELETE
...

@bpayt2
Copy link
Author

bpayt2 commented Mar 30, 2020

@ozankabak @gordiustechnologies Appreciate the second observation - I have tested your workaround and it does permit the filtering on event type (at the secondary level; event sensor). Thank you very much!

@VaibhavPage Understanding that it is not desirable to implement event type filtering within the event sensor and that we are still unable to leverage data filters within the event sensor, can you add this as an enhancement request? To reiterate; it would be desired to have a base64 decode option for event source's resource type and event sensors' filter.

@VaibhavPage
Copy link
Contributor

@gordiustechnologies Although the event data is base64 encoded when you use the parameters or filters, the sensor internally decodes the event data into a JSON format like following,

"data": {
          "type": "type_of_the_event", // ADD, UPDATE or DELETE
          "body": "resource_body", // JSON format
          "group": "resource_group_name",
          "version": "resource_version_name",
          "resource": "resource_name"
        }

The body is also base64 encode, but once you refer it in the sensor as I have done in the filter posted above, it decodes it into JSON format of the corresponding K8s object,

      filters:
        name: data-filter
        data:
          - path: body.metadata.name
            type: string
            value:
              - my-workflow

Node events of all types going through gateway should not be related to the event body being base64 encoded. I'll try to replicate it on my setup and check if there is an issue with node events.

@bpayt2 maybe I am not able to understand the problem, can you post your gateway, sensor and event source file and walk me through the desired outcome? It'll help me test on my local setup. Thanks!!

@ozankabak
Copy link

Thanks @VaibhavPage. Considering @bpayt2's posts, I suspect the event type filtering issue is not specific to nodes, but possibly affecting other resources too. Encoding may not be the root cause, but something is definitely interfering with the eventType filtering mechanism.

@VaibhavPage
Copy link
Contributor

VaibhavPage commented Mar 30, 2020

Got it. I have a pr #561 which will fix the issue if any. I'll keep you guys posted.

@ozankabak
Copy link

@VaibhavPage, can you make a 0.13.1 release so that we can verify whether eventType filtering works after this fix in our setups?

@VaibhavPage VaibhavPage reopened this Apr 1, 2020
@VaibhavPage
Copy link
Contributor

I'll cut a release in couple of days.

@ozankabak
Copy link

@VaibhavPage Does 0.14 fix this? Should we try it out in our setups?

@VaibhavPage
Copy link
Contributor

VaibhavPage commented Apr 13, 2020

@VaibhavPage
Copy link
Contributor

@ozankabak did it fix the issue?

@bpayt2
Copy link
Author

bpayt2 commented May 6, 2020

@VaibhavPage @ozankabak I have tested the latest and it fixes my reported issue. Please feel free to close. Thank you!

@bpayt2 bpayt2 closed this as completed May 6, 2020
@ozankabak
Copy link

@VaibhavPage: I can confirm that it works too. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants