feat(spans): Use gauges to report self and total time to lower costs #3448
Conversation
relay-dynamic-config/src/defaults.rs
Outdated
| }, | ||
| MetricSpec { | ||
| category: DataCategory::Span, | ||
| mri: "g:spans/self_time@millisecond".into(), |
There was a problem hiding this comment.
Can we stick with exclusive_time and duration instead of self_time and total_time? Or would that cause naming clashes in some downstream component that ignores the type prefix?
There was a problem hiding this comment.
I think we can stick with them but we do rename them to total_time and self_time in the UI. I thought that was a good occasion to stop having to do that.
Any reason why not to change the names?
There was a problem hiding this comment.
Is the purpose of this PR to experiment with what the product would look like without distributions, or is it already decided and we now need the double write for the transition period?
If it's just for experimenting, I don't think we actually need to record a gauge metric to see what the product would look like without distributions. Distributions are (almost) a superset of gauges, so we should be able to query min, max, sum, and count from the distributions table just as easily as from the gauges table.
|
This is not to experiment, this is to "downgrade" the metric to lower its cost since we only use averages and a solution with percentiles won't come by improving distributions. Gauges are cheaper than distributions to store. |
| ]; | ||
|
|
||
| if double_write_distributions_as_gauges { | ||
| metrics.append(&mut vec
We're currently storing self and total times for spans as distribution metrics. Those store a lot of information in order to be flexible enough to calculate percentiles.
Since querying percentiles has been too slow to display on any screen, we decided not to do it this way and use a different method. It's not necessary to keep storing them as distributions and we could use gauges instead to lower our cost.
The plan is to add those new gauge metrics, record them for a while, add support to query them in the product behind a feature flag and switch to them when we're happy with the result.