[exporter/datadog] Add support for custom list of attribute -> tag conversion

[carlreid]

Is your feature request related to a problem? Please describe.

While it's great that the Datadog exporter is mapping from the OpenTelemetry conventions, it seems that it is not possible to enrich telemetry with tags from the exporter on the collector.

Describe the solution you'd like

An option key/value pairing on the exporter to define an attribute -> tag mapping. It could be something like:

attributeTagMappings:
  - teamname: team
  - semver: version

Describe alternatives you've considered

It seems like this mapping could also be achieved on Logging Pipelines. Would be preferable to keep all the configuration in the one exporter definition for us however. Likewise, it may be that there is agreement to use one specific tag, but each sender would source it from a different attribute, and a pipeline would need to be configured for all of those.

Additional context

I'm not sure if these request should be on Datadog Agent, as that's where some changes will be required for this to be implement I guess.

[mx-psi]

Hi @carlreid, thanks for the issue, this is the right place to ask this if this is about the Datadog exporter. To help you further, I would need to know what telemetry signal is this about: is it about traces, metrics or logs? To be clear, you can probably do this already with the help of a processor, but I need a bit more info to be able to help you.

[carlreid]

Hey @mx-psi, great this is the right place then 😊

For us, we'd like anything that can be tagged, to have these tags. Though for now, this is more in focus on logs as my organization is currently only indexing logs that meet specific requirements, which includes certain non-standard (as in Datadog standard) tags.

We do already use processors (attributes and resource) to enrich with attributes such as deployment.environment, service.name and so on if they are missing. One of which, is a team attribute which I can see makes its way to Datadog on the log as an attribute, but in our case it needs to be a tag.

Do you see a way that a custom tag could be added via processors in this scenario?

[mx-psi]

👍 Thanks for details. I am going to ping @dineshg13 about this then for the logs-specific question.

As a side note and while you wait, since you have some interest about other types, I want to note that for metrics you can enable the metrics::resource_attributes_as_tags option to add all attributes verbatim to the metric as tags.

[dineshg13]

@carlreid if you set a attribute ddtags either in attributesprocessor or in filelogreciever like this https://github.com/DataDog/opentelemetry-helm-charts/blob/92f3484f60c3d084f79a2658f14bfc06b67679ef/charts/opentelemetry-demo-datadog/otelcol-config.yaml#L91 , it would be converted to tag on Datadog for logs signal

[carlreid]

Hey @dineshg13 🙂 I guess that could work.

Though it would likely mean we'd need to duplicate the log pipelines so that the ddtags attribute didn't make it to other exporters where the dd prefixed attributes don't make sense.

I guess I'll leave it up to you guys whether this should remain an open issue or not. I don't know enough if it should be a processor or exporter problem with this mapping when it comes to the OpenTelemtry set up 😁

[carlreid]

An update @dineshg13. I just tried:

Edit:

      transform/datadog:
        logs:
          statements:
            - set(attributes["ddtags"], Concat("", "team:", resource.attributes["team"])) where resource.attributes["team"] != nil

Since we already use the attributes processor to enrich the resource, then we can just pull the value from there, and run the transform/datadog processor as part of a new logs/datadog pipeline.

However, although I can see my tag, all the other tags that get pulled automatically like the kube_* related tags, version, and env have now vanished. Is there a way to combine the ddtags with these?

I guess setting the ddtags attribute is overriding the value set in the translator?

[dineshg13]

@carlreid Whats the order or transform & resourcedetection processor ? resourcedetection processor has to be last .

[carlreid]

@dineshg13 It's looking like:

        logs/datadog:
          receivers:
            - otlp
          processors:
            - resource/min-required-attributes
            - transform/datadog
            - memory_limiter
            - batch
          exporters:
            - datadog

I would need the resource processor first (not the resourcedetection one), as it would add the default attribute if it wasn't provided.

Then transform/datadog is described in my last message, which is the only place setting ddtags attribute. Without the transform/datadog the datadog exporter was happily picking up all the semconv attributes.

[dineshg13]

ye. you are right. This looks like a bug in the Datadog exporter. We will patch it soon