Otel / Open Telemetry for logs, metrics, traces and profiling

[hpvd]

couldn't find an issue regarding Otel support in Pinots repository,
https://github.com/apache/pinot/issues?q=is%3Aissue+otel

-> so I just wanted to give a starting point and some input for discussion :-)

With OpenTelemetry announces support for profiling with bidirectional linking to logs, metrics and traces
https://www.cncf.io/blog/2024/03/19/opentelemetry-announces-support-for-profiling/

and it is on the way to be suitable for low latency systems
progress on this is tracked in the parent issues within Apache Pulsar (which is the motivating use case)
apache/pulsar#21121

and since it's somehow becoming the go to standard with an ever-growing number of integration
see https://opentelemetry.io/ecosystem/integrations/

=> it may also be very interesting and valuable for Pinot...

[hpvd]

maybe also interesting:
Elastic offers to donated their eBPF-based continuous profiling agent to OTel
open-telemetry/community#1918

"We're introducing native correlation between continuous profiling signals and OTel traces, starting with Java."

this is the background (and ad) article on this, to show the use case:
"Pinpointing performance culprits with continuous profiling and distributed tracing correlation"
https://www.elastic.co/de/blog/continuous-profiling-distributed-tracing-correlation

Regarding time plan for expecting profiling in OTel:
"I would guess this to be rather a matter of several months (from March 27th, 2024) until we have a first, fully integrated version of the profiling agent / functionality (depending on the review result, the concrete realization plans, etc.)."
open-telemetry/community#1918 (comment)

[Jackie-Jiang]

cc @zhtaoxiang @xiangfu0

[PeterCorless]

For metrics, a related issue to watch is [Proposal] Generic Time Series Engine in Pinot #13760, which would add support for a time-series engine (PromQL, M3QL, etc.). So once you collected metrics through OTel agents and collectors, this would allow you to query them, visualize through Grafana integration, etc.