translate otel messaging.* to ecs

[stuartnelson3]

Motivation/summary

Translate relevant otel messaging.* semantic conventions to their
elastic-equivalent fields.

closes #5094

Checklist

• Update CHANGELOG.asciidoc
• Documentation has been updated

How to test these changes

Ingest messaging spans from an otel-instrumented messaging service.

EDIT: https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrumentation/github.com/Shopify/sarama/otelsarama/example

Check the transactions and spans in the kibana UI for the following:

transaction:

• transaction.type=messaging
• Message metadata has fields queue.name, system, and operation.

span:

• span.type=messaging
• span.subtype is the framework name, eg. kafka
• span.action is one of receive, send, or process
• Message metadata has fields queue.name.

[stuartnelson3]

Opening this to start discussion. I'm modeling this after prior art, but also adding specific transaction/span translation according to https://github.com/elastic/apm/blob/master/specs/agents/tracing-instrumentation-messaging.md, which the PR I was looking at did not have.

The questions I have are:

• Should I be doing the transaction/span translation
• Which of the fields from the message attributes should we be including in the Message object
• I'm inferring the send operation based on what I saw in the otel spec (linked in previous bullet), but I'm not sure if that's correct

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #5334 updated

• Start Time: 2021-05-31T09:26:47.111+0000

• Duration: 38 min 39 sec

• Commit: b85ed59

Test stats 🧪

Test	Results
Failed	0
Passed	6154
Skipped	120
Total	6274

Trends 🧪

[axw]

The big question for me is: when do we create a transaction or a span? At the moment we only create a transaction for root otel spans, or otel spans with span kind SERVER. I think we should probably also create transactions for span kind CONSUMER... but only in some cases? Like you might consume a message within a transaction, and in that case it should be a span.

[stuartnelson3]

I think we should probably also create transactions for span kind CONSUMER... but only in some cases? Like you might consume a message within a transaction, and in that case it should be a span.

Wouldn't the current logic hold then? Whether span type is SpanKindCONSUMER or not, it being the root span is still valid and will branch correctly between transaction and span.

[stuartnelson3]

It appears when consuming a message the sarama otel wrapper is trying to pull out a parent span ctx.

Messages sent using the sarama library will have a span context; preserving that relationship would be in line with what otel seems to be doing. If a producer isn't instrumented, but the consumer is, then we'll start a new transaction. Similarly, if a producer's span has no parent, it will also become a transaction.

My personal confusion at having a parent/child relationship connected by an async message bus notwithstanding, it seems like we should probably adhere to the "if it's a root span -> new tx; else new span" flow we currently have?

[axw]

It appears when consuming a message the sarama otel wrapper is trying to pull out a parent span ctx.

Messages sent using the sarama library will have a span context; preserving that relationship would be in line with what otel seems to be doing. If a producer isn't instrumented, but the consumer is, then we'll start a new transaction. Similarly, if a producer's span has no parent, it will also become a transaction.

According to our messaging spec:

• sending messages is always represented as a span within a transaction, e.g. a background task or within an HTTP request handler. For OpenTelemetry these spans get created even when not operating within another span, so in this case I think it's fine if we also create a transaction if there's no parent context. No changes needed for PRODUCER
• passive and active message reception leads to transactions and spans respectively

For the Sarama instrumentation you pointed at, we're only dealing with passive message reception: you register a callback and it's invoked whenever a new message is received. The "transaction" is the handling of a new message.

For JMS on the other hand, there are two ways of receiving messages: a "MessageListener" (passive like sarama's consumer) and "MessageConsumer" (active).

My personal confusion at having a parent/child relationship connected by an async message bus notwithstanding, it seems like we should probably adhere to the "if it's a root span -> new tx; else new span" flow we currently have?

Again following the spec (with a slight adjustment to allow root producer spans -> transactions):

• If it's a root OTel span, we should create a transaction.
• If it's a non-root producer, we should create a span.
• If it's a passively received/processed message, we should create a transaction.
• If it's an actively received message (i.e. something like a channel receive within an HTTP handler), we should create a span.

The challenge is how to distinguish between the last two bullet points. I'm not sure that we can do this with the information we have. I think we would need open-telemetry/opentelemetry-specification#366

[axw]

Verified with 7.14.0-BC2, using the opentelemetry-go-contrib sarama example:

Produce span type/subtype/action and properties:

Receive transaction type:

Receive transaction properties:

Consume span type/action properties:

The consume span is missing details, but that's because it's a custom span created by the example app without those details: https://github.com/open-telemetry/opentelemetry-go-contrib/blob/262f81f753760cd1333d1925918300be3f01e77e/instrumentation/github.com/Shopify/sarama/otelsarama/example/consumer/consumer.go#L87-L90