datadog: document resource_attributes_as_tags behaviour and tag mapping, link to docs for actual mappings

[ringerc]

Component(s)

exporter/datadog

Is your feature request related to a problem? Please describe.

The datadog exporter has a resource_attributes_as_tags option but it's lightly documented, and unclear what exactly its behaviour is.

      ## @param resource_attributes_as_tags - string - optional - default: false
      ## Set to true to add all resource attributes of a metric to its metric tags.
      ## When set to false, only a small predefined subset of resource attributes is converted
      ## to metric tags.
      #
      # resource_attributes_as_tags: false

Exactly what " small predefined subset of resource attributes" is converted to metric tags, and using what rules/mapping/logic?

Edit: This should link to https://docs.datadoghq.com/opentelemetry/guide/semantic_mapping/

Can this mapping be overridden/extended, or is it hard-coded?

Edit: It's hard coded in https://github.com/DataDog/opentelemetry-mapping-go/tree/main/pkg/otlp/attributes/attributes.go and is not extensible.

~~I've read the code and followed the breadcrumbs to https://github.com/DataDog/opentelemetry-mapping-go/blob/main/pkg/otlp/metrics/metrics_translator.go#L72 ResourceAttributesAsTags but as far as I can tell it doesn't actually do anything. ~~ Edit: This was because the DD platform offers no way to see the actual incoming metrics stream with individual datum tags so I couldn't tell that it'd stopped sending some tags for metrics.

I tripped over DataDog/opentelemetry-mapping-go#156 in the process of looking it up. Does it really just do nothing?

Edit: Yes, that code does nothing. resource_attributes_as_tags has an effect but it's implemented elsewhere, the code in opentelemetry-mapping-go is a stub that does nothing.

Describe the solution you'd like

Documentation on the mapping of resource attributes to tags, both with and without the resource_attributes_as_tags setting. Clearly linked and discoverable, and up to date.

Coverage of how this mapping may affect metric cardinality.

A means of overriding or extending the built-in mapping.

Describe alternatives you've considered

I've read the code and followed the breadcrumbs to https://github.com/DataDog/opentelemetry-mapping-go/blob/main/pkg/otlp/metrics/metrics_translator.go#L72 ResourceAttributesAsTags but as far as I can tell it doesn't actually do anything.

Some more digging revealed https://github.com/DataDog/opentelemetry-mapping-go/blob/main/pkg/otlp/attributes/attributes.go#L29 as the likely location of the hardcoded mapping of OpenTelemetry semantic conventions values to Datadog tags.

It'd really be better not to have to dig through the code to find this fairly fundamental info.

Additional context

Issues open on the mapping repo:

• Using resource attributes as tags in the dimensions DataDog/opentelemetry-mapping-go#156
• Otel resource attribute cloud.availability_zone mapped to "zone" tag not "availability-zone" tag DataDog/opentelemetry-mapping-go#214

This is made more confusing by there seeming to be no way to get the DD exporter to log what it really sends for metrics payloads: #29701

It looks like this mapping is not done for host_metadata discovery tags: #29700

[github-actions]

Pinging code owners:

• exporter/datadog: @mx-psi @gbbr @dineshg13 @liustanley @songy23 @mackjmr

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[ringerc]

I've since found that while the code in opentelemetry-mapping-go does appear to do absolutely nothing, there's a transform of resource attributes done in the exporter itself per DataDog/opentelemetry-mapping-go#156 (comment) . That's what actually does the work of the resource_attributes_as_tags setting.

Maybe the dead code should be removed or commented as a no-op?

There are datadog docs for the mapping of attributes too, they're just hard to find (not linked from other relevant docs), outdated and incomplete. See DataDog/opentelemetry-mapping-go#156 (comment) and particularly https://docs.datadoghq.com/opentelemetry/guide/semantic_mapping/ . The support team pointed these out to me; I didn't find them from reading other articles on the DD site.

[mackjmr]

Thanks for reporting @ringerc!

I've put up a PR to improve documentation: #29765

And PRs to remove the unused code:

• Remove ResourceAttributesAsTags DataDog/opentelemetry-mapping-go#219
• Remove call to metrics.WithResourceAttributesAsTags() DataDog/datadog-agent#21501
• Remove call to metrics.WithResourceAttributesAsTags() #29770

[ringerc]

Thanks very much for following up on it. Appreciated. I'd raised a docs PR to tweak some of the docs already, as linked above, but I didn't want to go through all of it.