Enhance data filter (argoproj#167)

* Fixing data filters issue * Provide ability to define multi-fields/multi-values data filter * Provide ability to define multi-fields/multi-values data filter
juliev0 · Feb 14, 2019 · b308e47 · b308e47
1 parent 296fb8f
commit b308e47
Show file tree

Hide file tree

Showing 18 changed files with 439 additions and 718 deletions.
diff --git a/docs/gateway-protocol.md b/docs/gateway-protocol.md
@@ -4,15 +4,12 @@
 ## Table of Contents
 
 - [github.com/argoproj/argo-events/pkg/apis/gateway/v1alpha1/generated.proto](#github.com/argoproj/argo-events/pkg/apis/gateway/v1alpha1/generated.proto)
-    - [EventProtocol](#github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.EventProtocol)
     - [Gateway](#github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.Gateway)
     - [GatewayList](#github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.GatewayList)
     - [GatewayNotificationWatcher](#github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.GatewayNotificationWatcher)
     - [GatewaySpec](#github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.GatewaySpec)
     - [GatewayStatus](#github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.GatewayStatus)
     - [GatewayStatus.NodesEntry](#github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.GatewayStatus.NodesEntry)
-    - [Http](#github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.Http)
-    - [Nats](#github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.Nats)
     - [NodeStatus](#github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.NodeStatus)
     - [NotificationWatchers](#github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.NotificationWatchers)
     - [SensorNotificationWatcher](#github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.SensorNotificationWatcher)
@@ -32,23 +29,6 @@
 
 
 
-<a name="github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.EventProtocol"></a>
-
-### EventProtocol
-Dispatch protocol contains configuration necessary to dispatch an event to sensor over different communication protocols
-
-
-| Field | Type | Label | Description |
-| ----- | ---- | ----- | ----------- |
-| type | [string](#string) | optional |  |
-| http | [Http](#github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.Http) | optional |  |
-| nats | [Nats](#github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.Nats) | optional |  |
-
-
-
-
-
-
 <a name="github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.Gateway"></a>
 
 ### Gateway
@@ -118,7 +98,7 @@ GatewaySpec represents gateway specifications
 | serviceSpec | [k8s.io.api.core.v1.Service](#k8s.io.api.core.v1.Service) | optional | ServiceSpec is the specifications of the service to expose the gateway Refer https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/#service-v1-core |
 | watchers | [NotificationWatchers](#github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.NotificationWatchers) | optional | Watchers are components which are interested listening to notifications from this gateway These only need to be specified when gateway dispatch mechanism is through HTTP POST notifications. In future, support for NATS, KAFKA will be added as a means to dispatch notifications in which case specifying watchers would be unnecessary. |
 | processorPort | [string](#string) | optional | Port on which the gateway event source processor is running on. |
-| eventProtocol | [EventProtocol](#github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.EventProtocol) | optional | EventProtocol is the underlying protocol used to send events from gateway to watchers(components interested in listening to event from this gateway) |
+| eventProtocol | [github.com.argoproj.argo_events.pkg.apis.common.EventProtocol](#github.com.argoproj.argo_events.pkg.apis.common.EventProtocol) | optional | EventProtocol is the underlying protocol used to send events from gateway to watchers(components interested in listening to event from this gateway) |
 
 
 
@@ -159,39 +139,6 @@ GatewayStatus contains information about the status of a gateway.
 
 
 
-<a name="github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.Http"></a>
-
-### Http
-
-
-
-| Field | Type | Label | Description |
-| ----- | ---- | ----- | ----------- |
-| port | [string](#string) | optional |  |
-
-
-
-
-
-
-<a name="github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.Nats"></a>
-
-### Nats
-
-
-
-| Field | Type | Label | Description |
-| ----- | ---- | ----- | ----------- |
-| url | [string](#string) | optional |  |
-| type | [string](#string) | optional |  |
-| clientId | [string](#string) | optional |  |
-| clusterId | [string](#string) | optional |  |
-
-
-
-
-
-
 <a name="github.com.argoproj.argo_events.pkg.apis.gateway.v1alpha1.NodeStatus"></a>
 
 ### NodeStatus

diff --git a/docs/sensor-protocol.md b/docs/sensor-protocol.md
@@ -6,15 +6,12 @@
 - [github.com/argoproj/argo-events/pkg/apis/sensor/v1alpha1/generated.proto](#github.com/argoproj/argo-events/pkg/apis/sensor/v1alpha1/generated.proto)
     - [ArtifactLocation](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.ArtifactLocation)
     - [ConfigmapArtifact](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.ConfigmapArtifact)
-    - [Data](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.Data)
     - [DataFilter](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.DataFilter)
+    - [DependencyGroup](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.DependencyGroup)
     - [EventDependency](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.EventDependency)
     - [EventDependencyFilter](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.EventDependencyFilter)
-    - [EventProtocol](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.EventProtocol)
     - [FileArtifact](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.FileArtifact)
     - [GroupVersionKind](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.GroupVersionKind)
-    - [Http](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.Http)
-    - [Nats](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.Nats)
     - [NodeStatus](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.NodeStatus)
     - [ResourceObject](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.ResourceObject)
     - [ResourceObject.LabelsEntry](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.ResourceObject.LabelsEntry)
@@ -28,6 +25,7 @@
     - [SensorStatus.NodesEntry](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.SensorStatus.NodesEntry)
     - [TimeFilter](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.TimeFilter)
     - [Trigger](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.Trigger)
+    - [TriggerCondition](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.TriggerCondition)
     - [URLArtifact](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.URLArtifact)
 
 
@@ -81,34 +79,35 @@ ConfigmapArtifact contains information about artifact in k8 configmap
 
 
 
-<a name="github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.Data"></a>
-
-### Data
+<a name="github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.DataFilter"></a>
 
+### DataFilter
+DataFilter describes constraints and filters for event data
+Regular Expressions are purposefully not a feature as they are overkill for our uses here
+See Rob Pike&#39;s Post: https://commandcenter.blogspot.com/2011/08/regular-expressions-in-lexing-and.html
 
 
 | Field | Type | Label | Description |
 | ----- | ---- | ----- | ----------- |
-| filters | [DataFilter](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.DataFilter) | repeated | filter constraints |
+| path | [string](#string) | optional | Path is the JSONPath of the event&#39;s (JSON decoded) data key Path is a series of keys separated by a dot. A key may contain wildcard characters &#39;*&#39; and &#39;?&#39;. To access an array value use the index as the key. The dot and wildcard characters can be escaped with &#39;\\&#39;. See https://github.com/tidwall/gjson#path-syntax for more information on how to use this. |
+| type | [string](#string) | optional | Type contains the JSON type of the data |
+| value | [string](#string) | repeated | Value is the allowed string values for this key Booleans are pased using strconv.ParseBool() Numbers are parsed using as float64 using strconv.ParseFloat() Strings are taken as is Nils this value is ignored |
 
 
 
 
 
 
-<a name="github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.DataFilter"></a>
+<a name="github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.DependencyGroup"></a>
 
-### DataFilter
-DataFilter describes constraints and filters for event data
-Regular Expressions are purposefully not a feature as they are overkill for our uses here
-See Rob Pike&#39;s Post: https://commandcenter.blogspot.com/2011/08/regular-expressions-in-lexing-and.html
+### DependencyGroup
+DependencyGroup is the group of dependencies
 
 
 | Field | Type | Label | Description |
 | ----- | ---- | ----- | ----------- |
-| path | [string](#string) | optional | Path is the JSONPath of the event&#39;s (JSON decoded) data key Path is a series of keys separated by a dot. A key may contain wildcard characters &#39;*&#39; and &#39;?&#39;. To access an array value use the index as the key. The dot and wildcard characters can be escaped with &#39;\\&#39;. See https://github.com/tidwall/gjson#path-syntax for more information on how to use this. |
-| type | [string](#string) | optional | Type contains the JSON type of the data |
-| value | [string](#string) | optional | Value is the expected string value for this key Booleans are pased using strconv.ParseBool() Numbers are parsed using as float64 using strconv.ParseFloat() Strings are taken as is Nils this value is ignored |
+| name | [string](#string) | optional | Name of the group |
+| dependencies | [string](#string) | repeated | Dependencies of events |
 
 
 
@@ -144,24 +143,7 @@ EventDependencyFilter defines filters and constraints for a event.
 | name | [string](#string) | optional | Name is the name of event filter |
 | time | [TimeFilter](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.TimeFilter) | optional | Time filter on the event with escalation |
 | context | [github.com.argoproj.argo_events.pkg.apis.common.EventContext](#github.com.argoproj.argo_events.pkg.apis.common.EventContext) | optional | Context filter constraints with escalation |
-| data | [Data](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.Data) | optional | Data filter constraints with escalation |
-
-
-
-
-
-
-<a name="github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.EventProtocol"></a>
-
-### EventProtocol
-EventProtocol contains configuration necessary to receieve an event from gateway over different communication protocols
-
-
-| Field | Type | Label | Description |
-| ----- | ---- | ----- | ----------- |
-| type | [string](#string) | optional | Type defines the type of protocol over which events will be receieved |
-| http | [Http](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.Http) | optional | Http contains the information required to setup a http server and listen to incoming events |
-| nats | [Nats](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.Nats) | optional | Nats contains the information required to connect to nats server and get subscriptions |
+| data | [DataFilter](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.DataFilter) | repeated | Data filter constraints with escalation |
 
 
 
@@ -201,45 +183,6 @@ to avoid automatic coercion.  It doesn&#39;t use a GroupVersion to avoid custom
 
 
 
-<a name="github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.Http"></a>
-
-### Http
-Http contains the information required to setup a http server and listen to incoming events
-
-
-| Field | Type | Label | Description |
-| ----- | ---- | ----- | ----------- |
-| port | [string](#string) | optional | Port on which server will run |
-
-
-
-
-
-
-<a name="github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.Nats"></a>
-
-### Nats
-Nats contains the information required to connect to nats server and get subscriptions
-
-
-| Field | Type | Label | Description |
-| ----- | ---- | ----- | ----------- |
-| url | [string](#string) | optional | URL is nats server/service URL |
-| startWithLastReceived | [bool](#bool) | optional | Subscribe starting with most recently published value. Refer https://github.com/nats-io/go-nats-streaming |
-| deliverAllAvailable | [bool](#bool) | optional | Receive all stored values in order. |
-| startAtSequence | [string](#string) | optional | Receive messages starting at a specific sequence number |
-| startAtTime | [string](#string) | optional | Subscribe starting at a specific time |
-| startAtTimeDelta | [string](#string) | optional | Subscribe starting a specific amount of time in the past (e.g. 30 seconds ago) |
-| durable | [bool](#bool) | optional | Durable subscriptions allow clients to assign a durable name to a subscription when it is created |
-| clusterId | [string](#string) | optional | The NATS Streaming cluster ID |
-| clientId | [string](#string) | optional | The NATS Streaming cluster ID |
-| type | [string](#string) | optional | Type of the connection. either standard or streaming |
-
-
-
-
-
-
 <a name="github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.NodeStatus"></a>
 
 ### NodeStatus
@@ -392,7 +335,9 @@ SensorSpec represents desired sensor state
 | dependencies | [EventDependency](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.EventDependency) | repeated | Dependencies is a list of the events that this sensor is dependent on. |
 | triggers | [Trigger](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.Trigger) | repeated | Triggers is a list of the things that this sensor evokes. These are the outputs from this sensor. |
 | deploySpec | [k8s.io.api.core.v1.PodSpec](#k8s.io.api.core.v1.PodSpec) | optional | DeploySpec contains sensor pod specification. For more information, read https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/#pod-v1-core |
-| eventProtocol | [EventProtocol](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.EventProtocol) | optional | EventProtocol is the protocol through which sensor receives events from gateway |
+| eventProtocol | [github.com.argoproj.argo_events.pkg.apis.common.EventProtocol](#github.com.argoproj.argo_events.pkg.apis.common.EventProtocol) | optional | EventProtocol is the protocol through which sensor receives events from gateway |
+| circuit | [string](#string) | optional | Circuit is a boolean expression of dependency groups |
+| dependencyGroups | [DependencyGroup](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.DependencyGroup) | repeated | DependencyGroups is a list of the groups of events. |
 
 
 
@@ -439,7 +384,7 @@ SensorStatus contains information about the status of a sensor.
 
 ### TimeFilter
 TimeFilter describes a window in time.
-Filters out event events that occur outside the time limits.
+DataFilters out event events that occur outside the time limits.
 In other words, only events that occur after Start and before Stop
 will pass this filter.
 
@@ -466,6 +411,24 @@ Trigger is an action taken, output produced, an event created, a message sent
 | resource | [ResourceObject](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.ResourceObject) | optional | Resource describes the resource that will be created by this action |
 | message | [string](#string) | optional | Message describes a message that will be sent on a queue |
 | replyStrategy | [RetryStrategy](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.RetryStrategy) | optional | RetryStrategy is the strategy to retry a trigger if it fails |
+| when | [TriggerCondition](#github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.TriggerCondition) | optional | When is the condition to execute the trigger |
+
+
+
+
+
+
+<a name="github.com.argoproj.argo_events.pkg.apis.sensor.v1alpha1.TriggerCondition"></a>
+
+### TriggerCondition
+TriggerCondition describes condition which must be satisfied in order to execute a trigger.
+Depending upon condition type, status of dependency groups is used to evaluate the result.
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| any | [string](#string) | repeated | Any acts as a OR operator between dependencies |
+| all | [string](#string) | repeated | All acts as a AND operator between dependencies |
 
 
 

diff --git a/examples/sensors/data-filter-webhook.yaml b/examples/sensors/data-filter-webhook.yaml
@@ -9,18 +9,19 @@ spec:
   deploySpec:
     containers:
       - name: "sensor"
-        image: "argoproj/sensor"
+        image: "argoproj/sensor:v0.7.1"
         imagePullPolicy: Always
     serviceAccountName: argo-events-sa
   dependencies:
-    - name: "webhook-gateway:foo"
+    - name: "webhook-gateway-http:foo"
       filters:
         name: "data-filter"
         data:
-          dataFilters:
-            - path: bucket
-              type: string
-              value: argo-workflow-input
+          - path: bucket
+            type: string
+            value:
+              - "argo-workflow-input"
+              - "argo-workflow-input1"
   eventProtocol:
     type: "HTTP"
     http:
@@ -33,15 +34,18 @@ spec:
         version: v1alpha1
         kind: Workflow
         source:
-          s3:
-            bucket:
-              name: workflows
-              key: hello-world.yaml
-            endpoint: minio-service.argo-events:9000
-            insecure: true
-            accessKey:
-              key: accesskey
-              name: artifacts-minio
-            secretKey:
-              key: secretkey
-              name: artifacts-minio
+          inline: |
+            apiVersion: argoproj.io/v1alpha1
+            kind: Workflow
+            metadata:
+              generateName: hello-world-
+            spec:
+              entrypoint: whalesay
+              templates:
+                - name: whalesay
+                  container:
+                    args:
+                      - "hello world"
+                    command:
+                      - cowsay
+                    image: "docker/whalesay:latest"
diff --git a/examples/sensors/time-filter-webhook.yaml b/examples/sensors/time-filter-webhook.yaml
@@ -21,10 +21,6 @@ spec:
         time:
           start: "22:14:05"
           stop: "23:04:05"
-          escalationPolicy:
-            name: "time-filter-escalation"
-            level: "Alert"
-            message: "time filter failed"
   triggers:
     - name: hello-world-workflow-trigger
       resource: