AMQP Gateway rate limit

[fayak]

Is your feature request related to a problem? Please describe.
In the project I'm using argo in, I often get a batch of ~800events sent by an application I can't easily modify. The problem is that our Kubernetes cluster isn't able to handle all of these events simultaneously, but the Argo AMQP Gateway doesn't have to logic to determine the maximum number of events it may grab. Therefore, it fetches all messages available on rabbitmq, and indirectly kills our cluster with heavy load.

I thought of limiting the number of pod in the cluster with Kubernetes, but all exceeding events leads to a failure.

Describe the solution you'd like
To this problem, I reckon 2 available solutions :

• Have an option in Argo to limit the number of simultaneous running workflow, and therefore have the gateway to not pull events from rabbitmq if it overflows this limit
• Have an option in the AMQP gateway to define a rate limit and enforce it. It may look like something as "max number of event pulled by x unit of time"

Describe alternatives you've considered
The options I've thought of were;

• max number of pods -> it make all exceeding workflows fail
• parallelism option -> It doesn't exist at the "namespace" level
• rate limit through rabbit -> Rabbit doesn't seems to have rate limits on exchanges
• rate limit in the sending application -> In our deployment it is really hard to add such feature
• have a third service pulling from an exchange used by the sending app, and pushing them to a second exchange where argo pulls events -> lol

Additional context

/

[VaibhavPage]

I may be wrong, but for option 1, you can take a look at parallelism level in Argo workflow. But I think it makes sense to have a rate limiter put in the gateway.

[fayak]

@VaibhavPage yes I've looked at parallelism, but, correct me if I'm wrong, it doesn't prevent the maximum simultaneous execution of workflows, but only steps (pods?) within a workflow

[VaibhavPage]

My bad, you are right.

[fayak]

Closing it as #378 got merged