Handle instrumentation scope in the Prometheus conversion spec

[dashpole]

Related: #1906

Background

OpenMetrics Naming+Namespacing

Changes

This adds "short_name" to the instrumentation scope, which would be an optional, single-word name for the scope. If present, it would be used as the metric prefix in Prometheus exporters. In prometheus receivers, the opentelemetry_instrumentation_scope metric would identify prefixes that can be removed from metrics, and used to reconstruct the Instrumentation scope in OTLP.

Alternatives:

• Turn the library name into the single-word metric prefix. E.g. goopentelemetryiocontribinstrumentationnethttpotelhttp_http_server_duration
  • Downside: This is ugly
• Add it as a label to all metrics within the scope: {"instrumentation_scope": "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"}
  • Downside: This isn't how OpenMetrics is designed to namespace metrics. Removing this later would be breaking for users.

cc @jmacd @Aneurysm9 @jsuereth

[MrAlias]

If the opentelemetry_instrumentation_scope info metric is present that that not be used for the instrumentation scope?

[dashpole]

Thats correct. It remains a separate metric stream to transmit that a given instrumentation library was used, but it doesn't allow you to figure out which metrics it applies to. I'm not sure it makes much sense, though, without the "short_name" in instrumentation scope. If we had that, then we could reconstruct the Instrumentation Scope here.

Maybe I should just skip this intermediary specification step, and just propose the "short_name" + round-tripping through prometheus...

[MrAlias]

Maybe I should just skip this intermediary specification step, and just propose the "short_name" + round-tripping through prometheus...

I'd be interested to hear what other think, but that sounds good to me.

[jmacd]

We should be careful to call the OpenMetrics "metric namespace" an equivalent of OTel's instrumentation library. I don't think this is completely true, see below.

Proposal for round-tripping Instrumentation Scope

I would like to see a treatment of instrumentation scope that could preserve the information for an OTel SDK exporting Prometheus data, which means a Prometheus scrape ideally would produce the same metrics as the SDK would have pushed. I think this is possible.

I've studied the OpenMetrics guidance on target info,

https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#supporting-target-metadata-in-both-push-based-and-pull-based-systems

and consider instrumentation scope information to be in a similar category. Suppose we use an OM info metric:

# TYPE otel_scope_info info
# HELP otel_scope_info Instrumentation scope metadata
otel_scope_info{name="some.dns.name/lib",version="0.1",schema_url="standard/url"} 1

To round-trip this correctly with an OTel scraper, an OTel SDK would specifically produce one library of metrics at a time. The OM/OTel translation would infer that metrics are organized by instrumentation scope such that the most recent otel_scope_info is implicitly associated with metrics in a scoped section. OTLP data scraped from other sources would leave instrumentation scope empty, as you propose.

Argument in favor of a first-class namespace concept

OpenMetrics writes about metric namespacing "the aim is to keep to a lightweight informal approach" and has three standard instrument constructor fields: "Namespace", "Subsystem", and "Name". With two optional underscore-separated fields and the potential for "Name" itself to contain underscores, there's no way to parse an OM metric name and infer what is namespace, what is subsystem, and what is name. Given no way to distinguish these, "metric namespace" appears to be a user-level option to ensure that when the same kind of instrumentation is produced by multiple distinct instances of the same instrumentation inside a single process, the user can force them not to aggregate or display in the same queries by adding an additional prefix. This would be done as-needed when the user is aware of conflicts and has the intention to solve them using namespaces, otherwise would not be done to keep metric names short.

To illustrate when metric namespace appears to solve for an OM user, imagine an OTel SDK configured with two clients speaking to two separate Redis instances with different load patterns, different kinds of data, different purposes. You've got two instances of the Redis client, and two instances of the OTel instrumentation.

Can the user have a way to distinguish the two scopes that works for both OM and OTel? IMO the user needs a first-class namespace concept here, which is not what OTel's instrumentation scope is at this time. Adding an additional property or property list to the Instrumentation Scope message type could help, and we could extend the OM<->OTel translation rules, for example, as follows. Suppose the MeterProvider has a new method called MeterProvider.Namespace(string) returning a MeterProvider where each Meter has the associated namespace. OTLP consumers would be advised to recognize this by prefixing metrics names in the section when displayed and queried.

This is what two instrumentation scopes for the same instrumentation library would look like when scraped, using the OM metric namespace:

# TYPE otel_scope_info info
# HELP otel_scope_info Instrumentation scope metadata
otel_scope_info{name="...",version="...",schema_url="...,namespace="firstprefix_"} 1

# TYPE firstprefix_http_requests
# HELP firstprefix_http_requests Number of HTTP requests
firstprefix_http_requests{} 1000

# TYPE otel_scope_info info
# HELP otel_scope_info Instrumentation scope metadata
otel_scope_info{name="...",version="...",schema_url="...,namespace="secondprefix_"} 1

# TYPE secondprefix_http_requests
# HELP secondprefix_http_requests Number of HTTP requests
secondprefix_http_requests{} 1000

[dashpole]

Unfortunately, I don't think the "Proposal for round-tripping Instrumentation Scope" complies with the OM spec: MetricFamilies MUST NOT be interleaved.

I haven't actually ever seen Namespace used to distinguish multiple instances of instrumentation--people just seem to use it as a second "subsystem" (e.g. in k8s), but maybe they were just using it wrong :). Since prom doesn't have a "View" concept, all metrics i've seen are fully named (including namespace, subsystem, etc.) by the library itself.

I think this is implied by the OM spec as well: the more public a library is the better namespaced its metric names should be implies that namespacing is performed by the library to keep it from colliding with other libraries.

So I think we are roughly on the same page, but need to figure out if the OM namespace is the equivalent of the instrumentation library, or if it is meant to be more than that.

@brian-brazil, we discussed this briefly at the prom WG yesterday. Can you help clarify the purpose of the OM namespace?

[brian-brazil]

Unfortunately, I don't think the "Proposal for round-tripping Instrumentation Scope" complies with the OM spec: MetricFamilies MUST NOT be interleaved.

I see no problems with the sample output. Though name and version are poor choices of label name that are highly likely to clash with something.

Can you help clarify the purpose of the OM namespace?

There's no formal concept of it in OM, as you say namespacing is performed by the library to keep it from colliding with other libraries.

[jmacd]

Unfortunately, I don't think the "Proposal for round-tripping Instrumentation Scope" complies with the OM spec: MetricFamilies MUST NOT be interleaved.

I see. A workaround would be to namespace the otel_scope_info section divider itself.

# TYPE firstprefix_otel_scope_info info
# HELP firstprefix_otel_scope_info Instrumentation scope metadata
firstprefix_otel_scope_info{name="...",version="...",schema_url="...,namespace="firstprefix_"} 1

# TYPE firstprefix_http_requests
# HELP firstprefix_http_requests Number of HTTP requests
firstprefix_http_requests{} 1000

# TYPE secondprefix_otel_scope_info info
# HELP secondprefix_otel_scope_info Instrumentation scope metadata
secondprefix_otel_scope_info{name="...",version="...",schema_url="...,namespace="secondprefix_"} 1

# TYPE secondprefix_http_requests
# HELP secondprefix_http_requests Number of HTTP requests
secondprefix_http_requests{} 1000

[brian-brazil]

OpenMetrics writes about metric namespacing "the aim is to keep to a lightweight informal approach" and has three standard instrument constructor fields: "Namespace", "Subsystem", and "Name".

OM has no such fields or notiions, that's what some client libraries did historically and I personally now consider it a bad idea. At the least it breaks grepability of metric names.

[dashpole]

I updated this to propose the short_name.

[jmacd]

This proposal from @bogdandrutu #2307 would give us a nice place to store the new "short_name" concept. I'm eager to see us add a list of arbitrary key:values to the scope, it would give us a way to distinguish two instances of the same instrumentation library which today we cannot do.

[tigrannajaryan]

I'm eager to see us add a list of arbitrary key:values to the scope, it would give us a way to distinguish two instances of the same instrumentation library which today we cannot do.

+1 to this. There are other use cases for this key:value list in the scope too, such as differentiating the type of data emitted from the scopes that belong to different data domains, e.g. profiling data emitted as log records or client-side data emitted as log records needs to be differentiated so that it can be easily routed and processed differently in the backends. We don't have a good way to handle this today.

[tigrannajaryan]

Given the comments about key-value lists in the Scope should the addition of ShortName be paused for now and instead we make it a semantic convention to be recorded in the key-value list?

[dashpole]

Yes. I'll close this for now, and will reopen after we've introduced the key-value list.

[joaopgrassi]

@tigrannajaryan @dashpole now that we have the scope attributes, should we bring this topic back, and define the semantic convention for scope attribute(s), where short_name would be the first?

[tigrannajaryan]

@tigrannajaryan @dashpole now that we have the scope attributes, should we bring this topic back, and define the semantic convention for scope attribute(s), where short_name would be the first?

Yes, feel free to make a proposal (or @dashpole feel free to do it).

We may need changes to semantic convention generator tooling to support scopes.

[joaopgrassi]

Unfortunately I don't have the cycles to work on this now, but I quickly tried it out and I think we don't need to change the convention generator.

Here's what I tried:
https://github.com/open-telemetry/opentelemetry-specification/compare/main...dynatrace-oss-contrib:opentelemetry-specification:feature/scope_attributes_semconv?expand=1

We could have a document inside specification/common/scope.md for the common scope attributes (short_name) for ex. Then, if there's scope attributes specific for each signal, those can be defined inside their appropriate folder like: semantic_conventions/metrics/scope-metrics.yaml -> specification/metrics/semantic_conventions/scope-metrics.md.

[tigrannajaryan]

@dashpole will you be able to work on this?

[dashpole]

Yes, I plan to work on this

[joaopgrassi]

Do we have an issue for it? Also, do you folks have an idea on how would we name this? We already have this page that states otel.scope.name and otel.scope.version. Would short_name also be mapped under the same? otel.scope.short_name? Or would it be under a new thing with just scope.short_name?

[tigrannajaryan]

Do we have an issue for it? Also, do you folks have an idea on how would we name this? We already have this page that states otel.scope.name and otel.scope.version. Would short_name also be mapped under the same? otel.scope.short_name? Or would it be under a new thing with just scope.short_name?

I don't think anything is decided yet. I would not assume otel.scope.* is the right approach. Should all scope attribute names start with otel.scope? Is there a reason the names cannot start from the root namespace?

[joaopgrassi]

Yeah I don't particularly have any answers, but was rather fishing for your ideas. I think we probably should have a new issue to discuss this topic further and lay out a good foundation for other attributes to come. I also don't think we need otel.scope.* as those are a bit "special".

[joaopgrassi]

I created an issue #2682 so we have a forum to discuss it. :)