Skip to content
This repository has been archived by the owner on Oct 9, 2023. It is now read-only.

Set unique labels string on prometheus metrics #136

Merged
merged 7 commits into from
Jun 9, 2022

Conversation

hamersaw
Copy link
Contributor

@hamersaw hamersaw commented Jun 7, 2022

TL;DR

This PR computes a unique labels set for each metric on initialization. This is used to popular metric values during increments (so a new slice does not have to be constructed), but more importantly it fixes an issue with corrupting the global metric keys slice.

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Complete description

The Golang append function implementation will reuse the underlying slice if it has remaining capacity. In the current implementation, this means that when we construct a new metric with additional labels we may modify the global metricKeys slice. This may result in the corruption of labels within other metrics.

Tracking Issue

fixes flyteorg/flyte#2597

Follow-up issue

NA

@hamersaw
Copy link
Contributor Author

hamersaw commented Jun 7, 2022

cc @honnix

@codecov
Copy link

codecov bot commented Jun 7, 2022

Codecov Report

Merging #136 (55e1ff8) into master (70688ae) will increase coverage by 0.68%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #136      +/-   ##
==========================================
+ Coverage   68.09%   68.78%   +0.68%     
==========================================
  Files          69       69              
  Lines        3401     3405       +4     
==========================================
+ Hits         2316     2342      +26     
+ Misses        913      904       -9     
+ Partials      172      159      -13     
Flag Coverage Δ
unittests 67.65% <100.00%> (+0.89%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
promutils/labeled/counter.go 81.25% <100.00%> (+22.91%) ⬆️
promutils/labeled/gauge.go 58.33% <100.00%> (+12.71%) ⬆️
promutils/labeled/stopwatch.go 80.00% <100.00%> (+1.05%) ⬆️
promutils/labeled/summary.go 83.33% <100.00%> (+2.38%) ⬆️
cache/auto_refresh.go 78.51% <0.00%> (+5.78%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70688ae...55e1ff8. Read the comment docs.

@hamersaw hamersaw changed the title Compute and set unique labels string on metrics Set unique labels string on prometheus metrics Jun 7, 2022
@honnix
Copy link
Member

honnix commented Jun 7, 2022

Shall we add at least one test case to cover the corruption problem?

EngHabu
EngHabu previously approved these changes Jun 7, 2022
Copy link
Contributor

@EngHabu EngHabu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for fixing this!

hamersaw added 4 commits June 7, 2022 16:55
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
hamersaw added 2 commits June 7, 2022 18:37
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
@hamersaw hamersaw requested a review from EngHabu June 7, 2022 23:48
@honnix
Copy link
Member

honnix commented Jun 9, 2022

Shall we get this merged and upgrade related components?

@EngHabu EngHabu merged commit 1279ea2 into master Jun 9, 2022
@hamersaw hamersaw deleted the bug/additional-labels-corrupt-metrics-keys branch June 9, 2022 16:00
eapolinario pushed a commit that referenced this pull request Sep 6, 2023
* computing and setting unique labels string on metrics

Signed-off-by: Daniel Rammer <[email protected]>

* added unit tests to counter

Signed-off-by: Daniel Rammer <[email protected]>

* added unit tests to gauge

Signed-off-by: Daniel Rammer <[email protected]>

* added unit tests to stopwatch

Signed-off-by: Daniel Rammer <[email protected]>

* added unit tests to summary

Signed-off-by: Daniel Rammer <[email protected]>

* fixed lint issue

Signed-off-by: Daniel Rammer <[email protected]>

* fixed another lint issue

Signed-off-by: Daniel Rammer <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Using additional labels in prometheus metrics can cause panic
3 participants