Multiple Kafka message keys/offsets as span attributes?

[Ocramius]

Area(s)

area:messaging

What's missing?

This is raised starting from open-telemetry/opentelemetry-go#6372

Context

I'm working on a messaging system that deals with message batches.

At pseudo-code level, messages are accumulated in a batch, then processed:

batcher := someBatcher(func(messages []Message, session KafkaSession) {
  handleBatch(messages) // I want to create a span around this

  for _, message := range messages {
    session.MarkProcessed(message)
  }
})

messageConsumer := makeConsumer(consumerGroupOptions, batcher); // this retrieves messages one-by-one, passes them to `batcher`

While trying to connect input messages and my processing, trace correlations get completely lost, because I may have thousands of input messages leading to a single trace, therefore diluting the concept of "one parent".

Describe the solution you'd like

Given a set of input Kafka messages with an identifier, we should be capable of starting a span that considers them some sort of "parent".

For example, RocketMQ has messaging.rocketmq.message.keys