Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extract consumer group member_count as a separate metric #103

Closed
amuraru opened this issue Jul 31, 2021 · 4 comments · Fixed by #105
Closed

Extract consumer group member_count as a separate metric #103

amuraru opened this issue Jul 31, 2021 · 4 comments · Fixed by #105

Comments

@amuraru
Copy link
Contributor

amuraru commented Jul 31, 2021 •
edited
Loading

Currently consumer group member_count is reported as a label in kminion_kafka_consumer_group_info metric.
When there is a high churn in a consumer group with members joining and leaving this leads to a high cardinality in this metric.
Moreover having a separate metric helps in monitoring the number of members in time.
amuraru added a commit to amuraru/kminion that referenced this issue Jul 31, 2021
@amuraru
Extract consumer group member_count as a separate metric
3c14d24 
Added a new consumer_group_members topic to capture number of members
in a consumer group and removed the `member_count` label from consumer_group_info

Fixes redpanda-data#103
@amuraru amuraru mentioned this issue Jul 31, 2021
Merged
amuraru added a commit to amuraru/kminion that referenced this issue Jul 31, 2021
@amuraru
Extract consumer group member_count as a separate metric
293fc8e 
Added a new consumer_group_members topic to capture number of members
in a consumer group and removed the `member_count` label from consumer_group_info

Fixes redpanda-data#103
amuraru added a commit to amuraru/kminion that referenced this issue Aug 1, 2021
@amuraru
Extract consumer group member_count as a separate metric
00ef547 
Added a new consumer_group_members topic to capture number of members
in a consumer group and removed the `member_count` label from consumer_group_info

Fixes redpanda-data#103
amuraru added a commit to amuraru/kminion that referenced this issue Aug 1, 2021
@amuraru
Extract consumer group member_count as a separate metric
06332a8 
Added a new consumer_group_members topic to capture number of members
in a consumer group and removed the `member_count` label from consumer_group_info

Fixes redpanda-data#103
amuraru added a commit to amuraru/kminion that referenced this issue Aug 1, 2021
@amuraru
Extract consumer group member_count as a separate metric
3450627 
Added a new consumer_group_members topic to capture number of members
in a consumer group and removed the `member_count` label from consumer_group_info

Fixes redpanda-data#103
@weeco
Copy link
Contributor

weeco commented Aug 1, 2021

I agree that this makes sense and I'm surprised I haven't done that from the beginning on. Thank you for raising this issue and also filing the PR. I will try to review the PRs in the course of next week. Feel free to ping me in case it takes longer

@amuraru
Copy link
Contributor Author

amuraru commented Aug 1, 2021

Thanks @weeco - sure, take your time.
similiar scalar labels exists in topic_info metric: partition_count and replication_factor (if enabled) albeit more stable. Wondering if we should do the same there

amuraru added a commit to amuraru/kminion that referenced this issue Aug 1, 2021
@amuraru
Extract consumer group member_count as a separate metric
d88a513 
Added a new consumer_group_members topic to capture number of members
in a consumer group and removed the `member_count` label from consumer_group_info

Fixes redpanda-data#103
@weeco
Copy link
Contributor

weeco commented Aug 2, 2021

Technically you are right about the two other metrics, but I think given the stability it's fine to keep them as label as part of the topic info metric.

@amuraru
Copy link
Contributor Author

amuraru commented Aug 2, 2021

Another disadvantage I've see with these numerical labels is when writing alert rules. I couldn't find a way create a prometheus rule that can check say: alert if number of any topic partitions go beyond 1000
Dashboarding is fine though as Grafana provides a way to chart extract and display label values

amuraru added a commit to amuraru/kminion that referenced this issue Aug 3, 2021
@amuraru
Extract consumer group member_count as a separate metric
3b14a4c 
Added a new consumer_group_members topic to capture number of members
in a consumer group and removed the `member_count` label from consumer_group_info

Fixes redpanda-data#103
amuraru added a commit to amuraru/kminion that referenced this issue Sep 20, 2021
@amuraru
Extract consumer group member_count as a separate metric
95b8407 
Added a new consumer_group_members topic to capture number of members
in a consumer group and removed the `member_count` label from consumer_group_info

Fixes redpanda-data#103
amuraru added a commit to amuraru/kminion that referenced this issue Sep 20, 2021
@amuraru
Extract consumer group member_count as a separate metric
4cc551d 
Added a new consumer_group_members topic to capture number of members
in a consumer group and removed the `member_count` label from consumer_group_info

Fixes redpanda-data#103
amuraru added a commit to amuraru/kminion that referenced this issue Sep 20, 2021
@amuraru
Extract consumer group member_count as a separate metric
ab8ef7c 
Added a new consumer_group_members topic to capture number of members
in a consumer group and removed the `member_count` label from consumer_group_info

Fixes redpanda-data#103
amuraru added a commit to amuraru/kminion that referenced this issue Oct 11, 2021
@amuraru
Extract consumer group member_count as a separate metric
00eb355 
Added a new consumer_group_members topic to capture number of members
in a consumer group and removed the `member_count` label from consumer_group_info

Fixes redpanda-data#103
amuraru added a commit to amuraru/kminion that referenced this issue Oct 23, 2021
@amuraru
Extract consumer group member_count as a separate metric
dd8009a 
Added a new consumer_group_members topic to capture number of members
in a consumer group and removed the `member_count` label from consumer_group_info

Fixes redpanda-data#103
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Extract consumer group member_count as a separate metric amuraru/kminion
2 participants
@amuraru @weeco