How to use Custom Collectors for Histograms

[cunningr]

When using a histogram metrics we can use the 'observe' method to evaluate which bucket our latest sample will fall in:

E.g.
h1.labels('device-1').observe(value.get('latest_sample'))
->

        for i, bound in enumerate(self._upper_bounds):
            if amount <= bound:
                self._buckets[i].inc(1)
                break

However as I am proxying metrics I believe I should be using a Customer Collector and the HistogramMetricFamily which only provides add_metric which seems to just populate all the buckets.

am I supposed to calculate the buckets in my code before writing the metric?

[mcbobke]

I have the same question. @brian-brazil is this how we are expected to build a Histogram in a custom collector?

[mcbobke]

Experimenting with this a bit, I was able to write a custom collector that utilizes the regular Histogram and not the HistogramMetricFamily with the following steps. All of this takes place in the Collect method of your custom collector.

1. Start with an empty list of metrics like metrics=[].
2. Create a Histogram with registry=None so that it is not registered to the default REGISTRY.
3. Add observations to the Histogram as you would outside of a custom collector.
4. Extend the metrics list with the return from histogram.collect().
5. Return the metrics list from the Collect method.

Example:

class CustomCollector(object):
    def __init__(self, namespace: str):
        self._namespace = namespace

    def collect(self):
        metrics = []

        hist = Histogram(
            name='http_response_time',
            documentation='HTTP response time',
            namespace=self._namespace,
            registry=None,
            labelnames=('method')
        )
        hist.labels('GET').observe(200)
        
        metrics.extend(hist.collect())
        return metrics

In your project's initialization code, ensure that the custom collector is registered.

I'm certain this is not supported, but implementing the Histogram's logic on my own sounds dreadful.

[brian-brazil]

That's how I'd do it, keep in mind that you possibly want to switch the type to GaugeHistogram before sending it on if you're using a custom collector like this.

[mcbobke]

Nice, glad I am on the right track. What is the benefit of using the GaugeHistogram?

[brian-brazil]

It's a type thing, so may be used by downstream tooling for better hints etc.. If the data is based on counters then it's a Histogram. If it's a snapshot of current state it's a GaugeHistogram.

[mcbobke]

Interesting, I've never seen a GaugeHistogram being used anywhere else. The use case I have is for buckets that increment (as with classic Histograms) so I think I'm safe to keep the current type, but is there any existing documentation or future plans to include documentation for this type? And maybe for other types like StatefulSet which I've never seen before looking through the source.

Appreciate the help!

[mcbobke]

Also @cunningr to keep a running count of the metrics that you're proxying (in the case of a classic Histogram), it wouldn't be a bad idea to use an instance variable with locking in the Collect method to ensure that multiple scrapes don't step on each other. In my example, I'm using gevent but you should use whatever library is appropriate to your project.

[brian-brazil]

You should create a new Histogram for each collect, otherwise you're risking race conditions.

[mcbobke]

Ah yes, my mistake. It's the job of Prometheus to collect the scrapes and store them within the Histogram. :)

[mcbobke]

Well.. actually, am I correct about that? For example, I have a Counter that is tracking the number of times a specific operation is performed. That counter is instrumenting this exporter that I'm writing, and therefore is not recreated on each scrape.

How do I ensure that this Histogram from the Custom Collector exposes buckets that are potentially incrementing on each /metrics call with new data from the external API?

[cunningr]

In that case, I guess you’re not proxying the data anymore? My use case is for reading counters from another system/format and exposing those as Prometheus metrics.

…

On Wed, 1 Sep 2021 at 20:54, Matthew Bobke ***@***.***> wrote: Well.. actually, am I correct about that? For example, I have a Counter that is tracking the number of times a specific operation is performed. That counter is instrumenting this exporter that I'm writing, and therefore is not recreated on each scrape. How do I ensure that a Histogram exposes buckets that are potentially incrementing on each /metrics call with new data from the external API? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#683 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHDBWJZHJRVNDO2WAGMPGNTT72AHXANCNFSM5APUTDWQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[brian-brazil]

In that case, manually putting together the Histogram's samples is what I'd do.

[csmarchbanks]

We (the Prometheus team) are trying out Github discussions so I moved this as it is not an issue with the library. Thanks all for the great answers.

[mcbobke]

Thanks for letting me know! I still have a couple of unanswered questions in this thread so hopefully it can stay active.

• Is there any future documentation planned for the metric types that are currently undocumented, namely the GaugeHistogram? I have not heard of this before today and have never seen it used elsewhere.
• I do not understand why I'd create a new Histogram in my Custom Collector on each scrape of the /metrics endpoint if my goal is to allow it to grow like a Histogram being used to instrument my own code. Excuse me if this is an ignorant question but if the reason is that it's prone to race conditions (which I understand) then how else would I achieve this?

[csmarchbanks]

I would happily accept a PR for more documentation on the metric types, or else could work on it after my current work project gets to a deadline in a month or so. You can find some documentation in the OpenMetrics spec, but I don't see a concise description of when to use each type.

As far as creating a new histogram each time it depends a bit on what the data coming from your data source looks like. If it is a bunch of counters then they will grow automatically (each counter will be higher) each time you scrape so creating a new histogram will naturally grow as well compared to the last one. Plus you will not need to worry about calculating differences between previous scrape values. If you are not getting counters from your source it might be necessary to keep a long term histogram or do something custom.