Gauge with no value?

[endotronic]

I think of a gauge as a momentary reading of something. A good example would be a value from a sensor. For this, imagine a thermometer. A gauge would be a good representation of the temperature from that thermometer. Well, what if the thermometer becomes unreadable? In this situation I would like to stop reporting the temperature at all, since it is now unknown. Keeping the last value set would misrepresent the situation and hide a condition that could be used for alerting.

I really like to use a callback with gauges (set_function) since this reflects how Prometheus works (e.g. read the thermometer when Prometheus scrapes the endpoint). It seems natural to raise an exception in this situation, if a value cannot be determined, but I find that this causes all metrics (even working ones) to go missing, as the HTTP endpoint begins to return "A server error occurred. Please contact the administrator." It should be possible to let one metric disappear while keeping others that still work intact.

Even using the gauge the "normal" way, I'm not sure how one would go about removing it. I imagine you cannot set the value to None, since returning None in the callback I just mentioned results in TypeError: float() argument must be a string or a number, not 'NoneType'.

Is there a way to temporarily remove a gauge?

[csmarchbanks]

Hello, the best solution is probably to use a Custom Collector to gather the temperature information and only create a gauge if there is a value. This keeps the behavior of reading the thermometer when Prometheus scrapes the endpoint, and allows you lots of control over the results.

It might also be possible to remove the gauge as needed, e.g. registry.unregister(temperature_gauge). Then re-register it when the value comes back. Keeping track of this state is probably more of a pain than using the custom collector though.

[endotronic]

Ah, Custom Collector looks great, thanks!