Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

List offsets for Stable consumer groups only #102

Closed
wants to merge 1 commit into from

Conversation

amuraru
Copy link
Contributor

@amuraru amuraru commented Jul 28, 2021

Reduce the number of metrics for clusters where lots are consumer groups are short-lived and empty.
The proposed change is to list the consumer group offsets only for Stable groups.

@weeco
Copy link
Contributor

weeco commented Jul 29, 2021

I think that's not desired for a lot of users as they use consumer group lags to monitor for potential issues (e.g. because the consuming application is no longer alive).

@amuraru
Copy link
Contributor Author

amuraru commented Jul 29, 2021

That can be a right usecase, I agree. Would it make sense to make this optional to define a list of group states for which thr offsets are reported? Default all.

again - in our environment there are lots of short lived cg and lots of low value metrics reported for them

@weeco
Copy link
Contributor

weeco commented Jul 29, 2021

Hmm I'm unsure.

I'd like to figure out what requests exactly take too long with the consumer groups. Maybe all issues are solved if you use the offsetsTopic as scrape mode? The largest clusters I tested against had ~400 consumer groups and it was okayish (3-5s request duration) for describing the consumer groups.

@amuraru
Copy link
Contributor Author

amuraru commented Jul 29, 2021

Agree - the scrape time is one dimension the other is the amount of metrics scraped by prometheus.

The overall time decreased when applying #101 patch so that would be something you could check please

let's give this PR more thought - I agree

@amuraru
Copy link
Contributor Author

amuraru commented Aug 3, 2021

Closing - will address this more generically in #108

@amuraru amuraru closed this Aug 3, 2021
@amuraru amuraru deleted the stable-cg-offsets branch October 23, 2021 21:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants