Enforce rate limiting

[hexedpackets]

Having a per-host rate limit will protect the rest of the metric pipeline against someone adding a bad metric.

There should be a metric when things are being rate-limited that way it can be monitored for.

Can do this guava - https://google.github.io/guava/releases/19.0/api/docs/index.html?com/google/common/util/concurrent/RateLimiter.html.

The limiting can be added where the current filter logic is done or as another filter itself (i think the latter is better).

[dmichel1]

Thinking about how this is deployed internally...

With GCE we can utilize puppet to allow teams to override the rate-limit.

With k8s we need to investigate if setting an environment variable will work with how the ffwd side car is injected with the admission controller. If it doesn't we might have to provide another mechanism to do so.

[hexedpackets]

With admission controllers, the whole container object is injected by the controller and can't be changed by the deployment.

We could use annotations on the pod, and have ffwd read those on startup if its running in k8s. That would also be extendable to setting limits on a namespace instead of every single pod.

[hexedpackets]

Instead of reading the k8s annotation in ffwd itself, we could use the downward API. It would require a bit more setup for someone using ffwd in kubernetes but not much and removes any issues around permissions for the kubernetes API.

For example, with an annotation of ffwd.spotify.com/rate: 1k the container can then be configured with

env:
- name: FFWD_RATE_LIMIT
  valueFrom:
    fieldRef:
      fieldPath: metadata.annotations['ffwd.spotify.com/rate']

This seems like a pretty good solution IMO. Any thoughts @dmichel1 ?

[dmichel1]

+1 it's really simple to customize for users.