ADR-42 consumer priority groups #279
Conversation
|
This is a draft, I have some work left to do here and some thinking but enough to get some feedback, i am not feeling too hot today but wanted to get this out, please discuss here. |
|
cc: @bruth lots of hope to get this in 2.11 |
| ```go | ||
| type PriorityGroupState struct { | ||
| Group string `json:"name"` | ||
| PinnedClientId string `json:"pinned_id,omitempty"` |
|
|
||
| * We have 1 group defined and all pulls have to belong to this group | ||
| * The policy is `pinned_client` that activates these behaviors | ||
| * When a pinned client has not done any pulls in the last 120 seconds the server will switch to another client |
There was a problem hiding this comment.
With this approach, longer expire interval, more sensitive the TTL becomes.
Maybe we should base TTL on Duration since last Pinned Client Pull Request expiry?
There was a problem hiding this comment.
Might be an option, my thinking was that people will tune this based on their usage, network, maintenance behaviours etc rather than have a lot of magic.
We could move to magic in a future version though but I am reluctant to start with magic.
There was a problem hiding this comment.
FWIW in my initial consumer group implementation I am using the idle timeout for the consumer (i.e. time since last pull request expired) and it's working for me. Otherwise what if you set the timeout to for example 60s and the applications are fetching with a timeout of more than 60s, wouldn't that lead to flapping when there are no new messages to deliver?
There was a problem hiding this comment.
Maybe not clear here but like the way it calculates if a consumer is idle or not it wouldn’t consider a client with an flight pull as being active.
But this seems open to bugs or abuse so perhaps you simply would have to ensure you pull more frequently. Alternatively we could cap the pull timeout so it relates to this timeout - no more than twice this timeout pull expire or something?
good point. Needs some thinking here
ripienaar
force-pushed
the
priority_group
branch
from
fe31831 to
c504b6b
Compare
ripienaar
force-pushed
the
priority_group
branch
from
c504b6b to
6a06327
Compare
ripienaar
changed the title
ripienaar
force-pushed
the
priority_group
branch
from
7250d66 to
1f81756
Compare
ripienaar
force-pushed
the
priority_group
branch
from
1f81756 to
b88e77d
Compare
Signed-off-by: R.I.Pienaar <[email protected]>
ripienaar
force-pushed
the
priority_group
branch
from
24baa04 to
3c19129
Compare
No description provided.