Make Tracing Great Again

[breezewish]

Feature Request

Is your feature request related to a problem? Please describe:

TiDB already supports TRACE statement, but it is rarely useful for several reasons:

1. It is not enabled by default due to slow. This limited its use cases. For example, it cannot be used to know how jitters happens.
2. It does not trace TiKV, missing a important piece.

Tracing is in fact a very useful piece of observability (log, metrics, trace). We did pretty well for log and metrics, but pretty bad for trace.

For now it is a bit hard to discover what causes a SQL to run slow in some scenarios. Existing facilities are:

• “EXPLAIN ANALYZE”: Only useful to know which executor takes a long time, cannot know RocksDB time, raft time, snapshot time, etc. Cannot inspect a SQL in the past.
• Tracing: As said before, only available in TiDB, cannot trace TiKV, and enabling it will greatly affect the performance so that it is rarely useful to inspect a SQL in the past.
• Metrics: It displays execution information of all SQLs, when payload is hybrid it is very difficult to figure out. Also metrics are not grouped in a way that we are easy to follow by SQL lifetime.
• Log / slow log: Different components output their own logs and it is very hard to link them together. Some logs only contain aggregated data and are not precise enough.

The new tracing implementation is supposed to provide diagnose capability for such scenarios.

Describe the feature you'd like:

Improve the tracing feature.

Stage 1.

1. Tracing should be efficient enough to be enabled by default for all SQL queries, and being stored for expansive or slow queries. In this way, we can know why a slow query happens. However, some user may care about performance a lot. In such case, we provide a global variable and/or session variable to disable it.
2. Tracing integrates spans from TiKV (and maybe PD) in one place.
3. The existing low performance tracing facility in TiDB shall be dropped.

The result of tracing can be further displayed in TiDB Dashboard in a nice way, for example, like what DataDog did:

Stage 2.

Users usually talk with executors instead of spans. The tracing should be able to be integrated with execution plans, allowing user to easily map each spans with executors, or providing a way to know why a specific executor is slow.

However overall tracing view is still necessary, since some spans do not belong to any executors.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

The tracing facility implementation in TiKV is already finished: https://github.com/pingcap-incubator/minitrace . It is super-efficient, can trace a span within 20ns. Even for shortest requests like point get, tracing 100 spans only introduces 6% performance lost (notice that in real life we are likely to trace < 10 spans for point get, so that the performance lost is negligible).

Tracing spans can be a data source for correspinding fields of slow log and metrics, to avoid counting duration repeatedly (and result in different durationos).

[SunRunAway]

cc @qw4990 I think it’s a considerable reimplementation for the entries in slow log.

[shenli]

It would be great if we can show the tracing result in DataDog. DataDog is popular and I heard from several users that they expect to see metrics or tracing in DataDog.
https://github.com/DataDog/dd-trace-go
The lack of official rust client may be an issue. I can only find this: https://github.com/pipefy/datadog-apm-rust

[zz-jason]

@breeswish @IANTHEREAL I moved it to the P1 priority, please confirm if that's correct.

[IANTHEREAL]

@zz-jason yes, do it