Determine impact of Prometheus UTF-8 support on OTel compatibility

[dashpole]

What are you trying to achieve?

Prometheus is adding UTF-8 support to its metrics format and promql query language, largely to be compatible with OpenTelmetry naming. This could possibly require breaking changes to the current prometheus specification, and we need to have a plan before marking the prometheus compatibility specification stable.

• Tracking Issue: UTF-8 character support for label and metric names  prometheus/prometheus#12630
• Proposal: Propose support for UTF-8 in metric and label names prometheus/proposals#28

Current OTel -> prometheus sanitization

The current OTel -> prometheus specification requires substituting underscores characters in names that do not match [a-zA-Z_:]([a-zA-Z0-9_:])*, and for characters in label keys that do not match [a-zA-Z_]([a-zA-Z0-9_])*

It also requires adding suffixes which use underscores, such as _seconds, or _total.

It also requires adding metrics, such as target_info, which are named with underscores, or labels, such as otel_scope_name.

Possible Resolutions

1. Decide to leave the specification as-is, and allow users to preserve the OTel name via config.
2. [breaking] Change the default to keep the OpenTelemetry names as-is, and allow users to revert to sanitized names via config.

Open Questions

• When preserving OTel names, should we also change suffixes, labels, and extra metrics to use dots instead of underscores? Otherwise, we would produce metrics like http.server.duration_seconds_total.

• If a Prometheus user wants to migrate from prometheus-style naming to OTel style naming, how would they accomplish that?

  • Could they use multiple queries in a single dashboard, or join the metrics with a query?
  • Could they use recording rules to align on a single name?
  • Could promql treat the names as equivalent so they don't need to do anything?

I think the resolution largely depends on what the migration looks like. If we chose to make a breaking change, it would have a massive impact, so the transition from current names to new names needs to be well thought out.

@open-telemetry/wg-prometheus @jsuereth

[jsuereth]

Some of my thoughts here:

While I hate saying this, I think figuring out a transition period where OTEL metric names are consistent within otel instrumentation, semantic conventions and prometheus makes a lot of sense. Yes this is a breaking change, but it's one we should very carefully engage with. (Note: I don't feel strongly we need to make this change, but this is the direction I'm leaning now)

If we accept the above, that would mean that we should likely move to . based separation for OTEL metrics and update our renaming rules appropriately.

I'd expect such a migration to take a LONG time and require a lot of tooling band-aides to actually accomplish it.

If we decide not to break things that exist today, I'm also ok with this, but I think we'll want to add a few additional tasks to the list you have:

• I'd like to have some kind of rendering of OpenTelemetry semantic conventions that denote what metrics look like in prometheus, in addition to what they look like in OpenTelemetry.
• We likely need to be more up-front with users that Unit matters when creating instruments for prometheus.

I'm curious what others think about these options.

[dashpole]

According to the Prometheus-wg notes from 11/8, @ywwg will be working on a migration plan, and will reach out when it is ready.

[dashpole]

Proposal: prometheus/proposals#30

[dashpole]

From the prometheus wg discussion on 12/6/23,

Prometheus exporters should support negotiation to determine if it should sanitize metric names and attributes. This will allow OpenTelemetry exporters to keep dots in names, but only if the server supports them.

Official prometheus client libraries will start allowing UTF-8 names, and will handle negotiation. OTel exporters implemented using these libraries will just have to update, and remove our sanitization. For OTel exporters that don't use prometheus client libraries, they will need to implement the negotiation to control whether they perform sanitization.

[tigrannajaryan]

@dashpole are you looking into this currently?

[dashpole]

Yes, i'm working on a survey + panel with the end-user SIG to try to drive a resolution to the naming questions above.