Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: mixed types of events in single EventSource #815

Merged
merged 8 commits into from
Aug 5, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion controllers/eventsource/validate.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ func ValidateEventSource(eventSource *v1alpha1.EventSource) error {
if rollingUpdates > 0 && recreates > 0 {
// We don't allow this as if we use recreate strategy for the deployment it will have downtime
eventSource.Status.MarkSourcesNotProvided("InvalidEventSource", "Some types of event sources can not be put in one spec")
return errors.New("event sources with rolling update and recreate update strategy can not put together")
return errors.New("event sources with rolling update and recreate update strategy can not be put together")
}

eventSource.Status.MarkSourcesProvided()
Expand Down
2 changes: 1 addition & 1 deletion controllers/eventsource/validate_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ func TestValidate(t *testing.T) {
testEventSource.Spec.Webhook = fakeWebhookEventSourceMap("test2")
err := ValidateEventSource(testEventSource)
assert.Error(t, err)
assert.Equal(t, "event sources with rolling update and recreate update strategy can not put together", err.Error())
assert.Equal(t, "event sources with rolling update and recreate update strategy can not be put together", err.Error())
})

t.Run("validate bad mixed types eventsource - duplicated name", func(t *testing.T) {
Expand Down
1 change: 1 addition & 0 deletions docs/assets/alpha.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
54 changes: 54 additions & 0 deletions docs/eventsource-deployment-strategies.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# EventSource Deployment Strategies

EventSource controller creates a k8s deployment for each EventSource object to
watch the events. Some of the event source types do not allow multiple live
clients with same attributes (i.e. multiple clients with same `clientID`
connecting to a NATS server), or multiple event source PODs will generate
duplicated events to downstream, so the deployment strategy and replica numbers
are different for different event sources.

## Rolling Update Strategy

`Rolling Update` strategy is applied to the following EventSource types:
VaibhavPage marked this conversation as resolved.
Show resolved Hide resolved

- AWS SNS
- AWS SQS
- Github
- Gitlab
- NetApp Storage GRID
- Slack
- Stripe
- Webhook

### Replicas Of Rolling Update Types

Deployment replica of these event source types respects `spec.replica` in the
EventSource object, defaults to 1.

## Recreate Strategy

`Recreate` strategy is applied to the following EventSource types:

- AMQP
- Azure Events Hub
- Kafka
- GCP PubSub
- File
- HDFS
- NATS
- Minio
- MQTT
- Emitter
- NSQ
- Pulsar
- Redis
- Resource
- Calendar

### Replicas Of Recreate Types

`spec.replica` in the `Recreate` types EventSources is ignored, the deployment
is always created with `replica=1`.

**Please DO NOT manually scale up the replicas, that will lead to unexpected
behaviors!**
41 changes: 41 additions & 0 deletions docs/eventsource-names.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# EventSource Names

In a Sensor object, a `dependency` is defined as:

```yaml
dependencies:
- name: test-dep
eventSourceName: webhook-example
eventName: example
```
The `eventSourceName` ad `eventName` might be confusing. Take the following
EventSource example, the `eventSourceName` and `eventName` are described as
below.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
name: webhook-example # eventSourceName
spec:
webhook:
example: # eventName
port: "12000"
endpoint: /example
method: POST
example-foo: # eventName
port: "13000"
endpoint: /example2
method: POST
```

## EventSourceName

`eventSourceName` is the `name` of the dependent `EventSource` object, i.e.
`webhook-example` in the example above.

## EventName

`eventName` is the map key of a configured event. In the example above,
`eventName` could be `exmaple` or `example-foo`.
78 changes: 78 additions & 0 deletions docs/eventsource-with-multiple-events.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# EventSource With Multiple Events

![alpha](assets/alpha.svg)

> v0.17.0 and after
Multiple events can be configured in a single EventSource, they can be either
one event source type, or mixed event source types with some limitations.

## Single EventSource Type

A single type EventSource configuration:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
name: webhook
spec:
webhook:
example:
port: "12000"
endpoint: /example
method: POST
example-foo:
port: "13000"
endpoint: /example2
method: POST
```
For the example above, there are 2 events configured in the EventSource named
`webhook`. Please use different `port` numbers for different events, this is the
limitation for multiple events configured in a `webhook` EventSource, this
limitation also applies to `webhook` extended event source types such as
`github`, `sns`.

## Mixed EventSource Types

EventSource is allowed to have mixed types of events configured.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
name: mixed-sources
spec:
webhook:
webhook-example: # eventName
port: "12000"
endpoint: /example
method: POST
sns:
sns-example: # eventName
topicArn: arn:aws:sns:us-east-1:XXXXXXXX:test
webhook:
endpoint: "/"
port: "15000"
accessKey:
key: my-key
name: my-name
secretKey:
key: my-secret-key
name: my-secret-name
region: us-east-1
```

However, there are some rules need to follow to do it:

- `Rolling Update` types and `Recreate` types can not be configured together,
see [EventSource Deployment Strategies](eventsource-deployment-strategies.md).

- Event Name (i.e. `webhook-example` and `sns-example` above, refer to
[EventSource Names](eventsource-names.md)) needs to be unique in the
EventSource, same `eventName` is not allowed even they are in different event
source types.

The reason for that is, we use `eventSourceName` and `eventName` as the
dependency attributes in Sensor.
7 changes: 7 additions & 0 deletions pkg/apis/common/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,14 +51,21 @@ var (
// RecreateStrategyEventSources refers to the list of event source types
// that need to use Recreate strategy for its Deployment
RecreateStrategyEventSources = []EventSourceType{
AMQPEvent,
CalendarEvent,
KafkaEvent,
PubSubEvent,
AzureEventsHub,
NATSEvent,
MQTTEvent,
MinioEvent,
EmitterEvent,
NSQEvent,
PulsarEvent,
RedisEvent,
ResourceEvent,
HDFSEvent,
FileEvent,
}
)

Expand Down