Allow PromQL-like aggregations with TraceQL (like in LogQL)

[ogxd]

Context

LogQL has support for some PromQL aggregation operations such as doing sum(rate({ app = "foo" }[1m])) * 2 (and many more kinds of operations https://grafana.com/docs/loki/latest/logql/). This, on top of the querying capabilities of LogQL, makes Loki very powerful.

Since TraceQL is "inspired by PromQL and LogQL", it would be very interesting to have the same kind of feature available.

Current solutions

Currently, the only two ways to aggregate trace information into time series data within Grafana are:

• Using transformations. This is very cumbersome, quite underperforming, and limited
  See related issue Rename Duration / Start time labels to prevent collision between traces and spans grafana#63740
• Generating Metrics from Spans. This is a convenient way to have some basic time series from traces, but as we can only select which labels to have for a predefined metric, cardinality quickly explodes and makes it not suited for many use cases.

Usecases

Some example use cases that can't be done (easily) today:

• Plot topk of trace errors per service instance
  Eg: topk(10, sum by (instance) (rate({ status = error })))
• Compute a rate of traces whose duration is above Xs
  Eg: sum(rate({ duration > 1s }[5m]))
• Get the average duration for calls to a given service by peer instance
  Eg (not sure about the syntax): avg_over_time({ peer.service = "redis" } | json | unwrap duration [5m]) by (instance)
  And even 95p (but it I guess would require Support creating histograms using logql loki#2953 to be resolved first)

Please let me know what you think of this. Maybe this is something is that already in the internal roadmap, I don't know, but I think it makes a lot of sense and would be an awesome feature for tempo.

[joe-elliott]

Thank you for this excellent write up. This is 100% on our mind as well. We are just beginning to spec out this exact functionality but on a reduced dataset and with a subset of TraceQL. These concessions were made to get this functionality into the hands of users as quickly as possible. We will try to provide more details as we solidify them and begin the work, but I'd also encourage you to join the monthly community calls where this will be a growing topic of conversation:

https://docs.google.com/document/d/1yGsI6ywU-PxZBjmq3p3vAXr9g5yBXSDk4NU8LGo8qeY/edit

The current functionality we are looking to deliver is something like:

• Only on recent data (past hour or so)
• spans with span.kind = server (all http, grpc, etc)
• a subset of traceql. perhaps something like binary operations in a single spanset { <any combination of parameters here> }

Don't hold me to this! it's a work in progress. This is along the lines of what we're thinking about for "phase 1" of this functionality. Like I said this will be an active discussion in the next few community calls. Also feel free to give feedback here!

[ogxd]

Awesome! Thank you for the link. I can't wait to see and hopefully try this 😄

[ogxd]

UPDATE: Tempo 2.4 and 2.5 aggregations are awesome! Thanks a lot