[Question] Opentelemetry format for resource.attributes (replacing dot with at sign)

[olia-dev]

Currently a log message in dataprepper looks like this, after being ingested via the otel-logs-source:

{"traceId":"","spanId":"","severityText":"","flags":0,"time":"2024-06-18T10:11:45.351322226Z","severityNumber":0,"droppedAttributesCount":0,"serviceName":null,"body":"time=\"2024-06-18T10:11:45Z\" level=info msg=\"Reconciliation completed\" application=observability/grafana dedup_ms=0 dest-name=in-cluster dest-namespace=observability dest-server=\"https://kubernetes.default.svc\" diff_ms=105 fields.level=1 git_ms=806 health_ms=0 live_ms=1 patch_ms=0 setop_ms=0 settings_ms=0 sync_ms=0 time_ms=1007","observedTime":"2024-06-18T10:11:45.438489611Z","schemaUrl":"https://opentelemetry.io/schemas/1.6.1","resource.attributes.source@environment":"dev","log.attributes.time":"2024-06-18T10:11:45.351322226Z","resource.attributes.k8s@namespace@name":"argocd","resource.attributes.cloud@account@id":"1234","resource.attributes.k8s@container@name":"application-controller","log.attributes.logtag":"F","resource.attributes.host@name":"ip-172-17-8-50.eu-central-1.compute.internal","resource.attributes.host@id":"i-1234","resource.attributes.k8s@pod@start_time":"2024-06-17T10:43:03Z","resource.attributes.cloud@region":"eu-central-1","log.attributes.log@iostream":"stderr","resource.attributes.host@image@id":"ami-1234","resource.attributes.source@project":"1234","resource.attributes.k8s@pod@name":"argocd-application-controller-0","log.attributes.log@file@path":"/var/log/pods/argocd_argocd-application-controller-0_0b9c78a1-2fed-4548-bd0f-f80ffe67faba/application-controller/0.log","resource.attributes.k8s@pod@uid":"0b9c78a1-2fed-4548-bd0f-f80ffe67faba","resource.attributes.k8s@node@name":"ip-172-17-8-50.eu-central-1.compute.internal","resource.attributes.cloud@platform":"aws_ec2","resource.attributes.k8s@container@restart_count":"0","resource.attributes.k8s@cluster@name":"1234","resource.attributes.ec2@tag@karpenter@sh/nodepool":"fallback-spot","resource.attributes.cloud@availability_zone":"eu-central-1a","resource.attributes.host@type":"c6gn.xlarge","resource.attributes.cloud@provider":"aws"}

The log itself was collected via otel/reciever/filelog.

According to the OpenTelemetry documentation, for example the resource.attributes.cloud should contain keys that are separted by a dot, instead of the @ sign.
Instead they seem to be replaced here.

Is there a reason, that the keys under resource.attributes and log.attributes are separated with @ instead of being kept in its original (Otel) format?

[oeyh]

@KarstenSchnitter Can you help take a look at this issue?

[KarstenSchnitter]

This "dedotting" is necessary because OpenSearch interprets dots in field names as nested JSON objects. I wrote a longer explanation of this in #4977. This process helps avoiding field type conflicts during indexing.