Why are histograms and counters not automatically deleted but gauges are?

[samuelclay]

I'm having an issue with increasing scrape durations on the /metrics endpoint and I believe it can be solved by automatically clearing out my data directory (PROMEHTEUS_MULTIPROC_DIR) on gunicorn worker reloads. It will clear on deploys, but I don't deploy everyday and the scrape duration goes up pretty quickly, esp. for a set number of urls.

Having read through #275 by way of #337, I found that mark_process_dead does clear out the data directory but only for gauges.

https://github.com/prometheus/client_python/blob/master/prometheus_client/multiprocess.py#L155-L162

Why are histograms and counters not automatically deleted but gauges are?

[csmarchbanks]

Great question.

The reason that histograms and counters are not automatically deleted is due to how Prometheus handles counter resets in functions such as rate. When Prometheus sees any decrease in a counter it interprets that as meaning the counter went all the way to zero, and back to the current value, which works great on service restarts. However, when mark_process_dead is called it only deletes the metrics for its PID, which would cause a partial reset of any counters if counters were deleted. A partial reset will look like a very large spike to Prometheus, for example a partial reset from 1000 -> 900 then increments to 950 would look like 950 increments occurred when in reality only 50 did. The only way to safely reset counters is to reset every counter to 0 for every PID.

Let me know if that didn't make sense/you have any follow up questions!

[samuelclay]

The only way to safely reset counters is to reset every counter to 0 for every PID.

Is there a way to do this not at application deployment? I have a directory truncation running on gunicorn reload, and that works fine, but how do I clear everything when any of the histogram db files could be written to?

In other words, I cleared out the directory out from underneath the running app and it threw a missing file exception, right in the same code that references which gauge or counter to delete. How do I avoid that exception? I'm happy to reset absolute counts, since grafana shows them as continuous rates anyhow.

[samuelclay]

This is the graph I want to flatline.

[csmarchbanks]

There is no built in way to safely do this today. #568 is the issue tracking this behavior, and there was some work attempted on this, but I think it is all stale at this point. It would be great to have a contribution to fix this issue, but there are lots of race conditions to consider so it is not easy.