OpenTelemetry span documents are missing the transaction.id field

[cyrille-leclerc]

ESS / APM Server version (apm-server version):

8.2.0

Description of the problem including expected versus actual behavior:

As identified with @gbamparop , span documents generated by the OpenTelemetry Java SDK 1.13.0 are missing the transaction.id field. It's likely to be a problem for all OpenTelemetry SDKs, not specific to the Java SDK, it's likely to be a missing enrichment of OpenTelemetry originated span documents in Elastic APM Server.

OpenTelemetry originated span document

Expand to see OpenTelemetry originated span document

```json { "_index": ".ds-traces-apm-default-2022.05.08-000003", "_id": "ozFgGYEBURctJ-VXembL", "_version": 1, "_score": 1, "_source": { "agent": { "name": "opentelemetry/java", "version": "1.13.0" }, "data_stream.namespace": "default", "processor": { "name": "transaction", "event": "transaction" }, "data_stream.type": "traces", "labels": { "ci_pipeline_run_cause": [ "UserIdCause:admin" ], "ci_pipeline_type": "multibranch", "ci_pipeline_run_completed": "true", "ci_pipeline_run_url": "http://localhost:8080/job/ecommerce-antifraud/job/main/171/", "jenkins_version": "2.332.3", "service_namespace": "jenkins", "exception_message": "PIPELINE_FAILURE", "ci_pipeline_id": "ecommerce-antifraud/main", "process_runtime_description": "AdoptOpenJDK OpenJDK 64-Bit Server VM 11.0.11+9", "ci_pipeline_run_result": "FAILURE", "ci_pipeline_name": "ecommerce-antifraud » main", "jenkins_url": "http://localhost:8080/", "exception_type": "PIPELINE_FAILURE", "jenkins_opentelemetry_plugin_version": "2.6.0", "ci_pipeline_multibranch_type": "branch" }, "observer": { "hostname": "dc2df1371625", "id": "eaf3dcb9-81a3-41b5-94a4-b820fd322d9e", "ephemeral_id": "1a6a6005-e783-4147-95ff-6daf61d120fe", "type": "apm-server", "version": "8.2.0" }, "trace": { "id": "eb8f142a8ac992c5b03f21c8c654b752" }, "@timestamp": "2022-05-31T09:08:17.387Z", "ecs": { "version": "1.12.0" }, "numeric_labels": { "ci_pipeline_run_number": 171, "ci_pipeline_run_durationMillis": 59884 }, "service": { "node": { "name": "MacBook-Pro.localdomain" }, "framework": { "name": "io.jenkins.opentelemetry", "version": "2.6.0" }, "name": "jenkins", "runtime": { "name": "OpenJDK Runtime Environment", "version": "11.0.11+9" }, "language": { "name": "java" }, "version": "2.332.3" }, "data_stream.dataset": "apm", "host": { "hostname": "MacBook-Pro.localdomain", "os": { "type": "macos", "platform": "darwin", "full": "Mac OS X 11.4" }, "name": "MacBook-Pro.localdomain", "architecture": "x86_64" }, "event": { "agent_id_status": "missing", "ingested": "2022-05-31T09:09:22Z", "outcome": "failure" }, "transaction": { "result": "Error", "duration": { "us": 60307244 }, "name": "BUILD ecommerce-antifraud/main", "id": "ebfec863b0a832e0", "type": "job", "sampled": true }, "timestamp": { "us": 1653988097387278 } }, "fields": { "transaction.name.text": [ "BUILD ecommerce-antifraud/main" ], "labels.ci_pipeline_run_completed": [ "true" ], "labels.process_runtime_description": [ "AdoptOpenJDK OpenJDK 64-Bit Server VM 11.0.11+9" ], "labels.ci_pipeline_run_url": [ "http://localhost:8080/job/ecommerce-antifraud/job/main/171/" ], "host.os.full": [ "Mac OS X 11.4" ], "labels.jenkins_url": [ "http://localhost:8080/" ], "host.hostname": [ "MacBook-Pro.localdomain" ], "service.language.name": [ "java" ], "transaction.result": [ "Error" ], "numeric_labels.ci_pipeline_run_durationMillis": [ 59884 ], "transaction.id": [ "ebfec863b0a832e0" ], "processor.event": [ "transaction" ], "agent.name": [ "opentelemetry/java" ], "host.name": [ "MacBook-Pro.localdomain" ], "event.agent_id_status": [ "missing" ], "event.outcome": [ "failure" ], "host.os.type": [ "macos" ], "processor.name": [ "transaction" ], "transaction.duration.us": [ 60307244 ], "labels.ci_pipeline_multibranch_type": [ "branch" ], "service.runtime.version": [ "11.0.11+9" ], "data_stream.type": [ "traces" ], "host.architecture": [ "x86_64" ], "timestamp.us": [ 1653988097387278 ], "observer.id": [ "eaf3dcb9-81a3-41b5-94a4-b820fd322d9e" ], "labels.ci_pipeline_run_result": [ "FAILURE" ], "labels.ci_pipeline_name": [ "ecommerce-antifraud » main" ], "observer.version": [ "8.2.0" ], "ecs.version": [ "1.12.0" ], "observer.type": [ "apm-server" ], "transaction.name": [ "BUILD ecommerce-antifraud/main" ], "agent.version": [ "1.13.0" ], "labels.jenkins_version": [ "2.332.3" ], "service.framework.version": [ "2.6.0" ], "service.node.name": [ "MacBook-Pro.localdomain" ], "labels.ci_pipeline_run_cause": [ "UserIdCause:admin" ], "transaction.sampled": [ true ], "trace.id": [ "eb8f142a8ac992c5b03f21c8c654b752" ], "labels.ci_pipeline_id": [ "ecommerce-antifraud/main" ], "labels.ci_pipeline_type": [ "multibranch" ], "service.name": [ "jenkins" ], "service.framework.name": [ "io.jenkins.opentelemetry" ], "data_stream.namespace": [ "default" ], "labels.exception_type": [ "PIPELINE_FAILURE" ], "service.runtime.name": [ "OpenJDK Runtime Environment" ], "observer.hostname": [ "dc2df1371625" ], "labels.jenkins_opentelemetry_plugin_version": [ "2.6.0" ], "transaction.type": [ "job" ], "event.ingested": [ "2022-05-31T09:09:22.000Z" ], "@timestamp": [ "2022-05-31T09:08:17.387Z" ], "service.version": [ "2.332.3" ], "observer.ephemeral_id": [ "1a6a6005-e783-4147-95ff-6daf61d120fe" ], "host.os.platform": [ "darwin" ], "labels.exception_message": [ "PIPELINE_FAILURE" ], "data_stream.dataset": [ "apm" ], "numeric_labels.ci_pipeline_run_number": [ 171 ], "labels.service_namespace": [ "jenkins" ] } } ```

Elastic APM Agent originated span document

Expand to see Elastic APM Agent originated span document

```json { "_index": "apm-7.11.0-span-000001", "_id": "omDtZncBSWNVmff8N00E", "_version": 1, "_score": 1, "_source": { "container": { "id": "cbc75e38c96c10ca9d7286efe10c43ee3525d57012a3a88fb6b01a811734c782" }, "parent": { "id": "21e4790e3944bd97" }, "kubernetes": { "pod": { "uid": "ed95bee8-0f3c-4c46-9318-97b93881dbee", "name": "checkoutservice-7b89d8664d-wjsdc" } }, "process": { "args": [ "/checkoutservice" ], "pid": 1, "title": "checkoutservice", "ppid": 0 }, "agent": { "name": "go", "version": "1.9.0" }, "processor": { "name": "transaction", "event": "span" }, "cloud": { "availability_zone": "us-central1-a", "instance": { "name": "gke-eden-3-staging-ssd-3-bc7684a4-li6q", "id": "6196144756140899491" }, "provider": "gcp", "machine": { "type": "n1-standard-16" }, "project": { "name": "elastic-product", "id": "150691754250" }, "region": "us-central1" }, "observer": { "hostname": "7bb22632d23b", "name": "instance-0000000000", "id": "37a442d8-1168-4b83-8292-d13053cc9d3f", "type": "apm-server", "ephemeral_id": "9f20e196-c0ce-4167-a059-11aa5b74536c", "version": "7.11.0", "version_major": 7 }, "trace": { "id": "0b225e23d0a705897cefcde4702d7e43" }, "@timestamp": "2021-02-03T08:05:57.847Z", "ecs": { "version": "1.6.0" }, "service": { "node": { "name": "cbc75e38c96c10ca9d7286efe10c43ee3525d57012a3a88fb6b01a811734c782" }, "environment": "prod", "name": "checkoutService", "runtime": { "name": "gc", "version": "go1.12.17" }, "language": { "name": "go", "version": "go1.12.17" }, "version": "1.6" }, "host": { "os": { "platform": "linux" }, "ip": "35.226.16.120", "architecture": "amd64" }, "event": { "outcome": "unknown" }, "transaction": { "id": "ad73184f70fa09d1" }, "span": { "duration": { "us": 4637 }, "subtype": "grpc", "name": "/hipstershop.CartService/GetCart", "destination": { "service": { "resource": "cartservice:11000", "name": "grpc", "type": "external" } }, "id": "44bb66f578604c26", "type": "external" }, "timestamp": { "us": 1612339557847919 } }, "fields": { "process.pid": [ 1 ], "service.language.name": [ "go" ], "cloud.availability_zone": [ "us-central1-a" ], "container.id": [ "cbc75e38c96c10ca9d7286efe10c43ee3525d57012a3a88fb6b01a811734c782" ], "transaction.id": [ "ad73184f70fa09d1" ], "processor.event": [ "span" ], "agent.name": [ "go" ], "event.outcome": [ "unknown" ], "cloud.region": [ "us-central1" ], "process.ppid": [ 0 ], "processor.name": [ "transaction" ], "service.runtime.version": [ "go1.12.17" ], "span.id": [ "44bb66f578604c26" ], "span.type": [ "external" ], "host.architecture": [ "amd64" ], "timestamp.us": [ 1612339557847919 ], "cloud.provider": [ "gcp" ], "cloud.machine.type": [ "n1-standard-16" ], "observer.id": [ "37a442d8-1168-4b83-8292-d13053cc9d3f" ], "observer.version": [ "7.11.0" ], "observer.type": [ "apm-server" ], "ecs.version": [ "1.6.0" ], "parent.id": [ "21e4790e3944bd97" ], "agent.version": [ "1.9.0" ], "span.destination.service.name": [ "grpc" ], "process.title": [ "checkoutservice" ], "span.destination.service.type": [ "external" ], "span.name": [ "/hipstershop.CartService/GetCart" ], "observer.name": [ "instance-0000000000" ], "service.node.name": [ "cbc75e38c96c10ca9d7286efe10c43ee3525d57012a3a88fb6b01a811734c782" ], "kubernetes.pod.uid": [ "ed95bee8-0f3c-4c46-9318-97b93881dbee" ], "cloud.project.name": [ "elastic-product" ], "host.ip": [ "35.226.16.120" ], "cloud.instance.id": [ "6196144756140899491" ], "trace.id": [ "0b225e23d0a705897cefcde4702d7e43" ], "span.duration.us": [ 4637 ], "kubernetes.pod.name": [ "checkoutservice-7b89d8664d-wjsdc" ], "service.environment": [ "prod" ], "service.name": [ "checkoutService" ], "service.runtime.name": [ "gc" ], "process.args": [ "/checkoutservice" ], "span.subtype": [ "grpc" ], "observer.version_major": [ 7 ], "observer.hostname": [ "7bb22632d23b" ], "@timestamp": [ "2021-02-03T08:05:57.847Z" ], "service.version": [ "1.6" ], "observer.ephemeral_id": [ "9f20e196-c0ce-4167-a059-11aa5b74536c" ], "host.os.platform": [ "linux" ], "service.language.version": [ "go1.12.17" ], "span.destination.service.resource": [ "cartservice:11000" ], "cloud.project.id": [ "150691754250" ], "cloud.instance.name": [ "gke-eden-3-staging-ssd-3-bc7684a4-li6q" ] } } ```

[axw]

This is expected. OpenTelemetry does not provide this information on span events. The UI should not assume transaction.id is always present.

[axw]

I'm going to close this as the server is working as expected. Let's discuss in a UI issue what the options are. We can always reopen this, but I can't currently see a viable option to implement it.

[LucaWintergerst]

I already have an UI issue open for this here: elastic/kibana#131050