Supportability: Revisit the metrics we have

[jpkrohling]

Context: #2098 (comment)

This issue here is to track which metrics we have in the usual case, which ones are shown when a problem happens, and what we could be having to help identify problems.

[jpkrohling]

Please, assign this to me.

[jpkrohling]

General metrics that every receiver/exporter should have

otelcol_receiver_accepted_spans
otelcol_receiver_refused_spans

otelcol_exporter_sent_spans
otelcol_exporter_send_failed_spans

OTLP-specific

In the receiver, the metric contains the protocol that received the batch:

otelcol_receiver_accepted_spans{receiver="otlp",service_instance_id="...",transport="grpc"} 2

No special metrics for the exporter.

Jaeger-specific

In the receiver, the metric contains the protocol that received the batch:

otelcol_receiver_accepted_spans{receiver="jaeger",service_instance_id="...",transport="udp_thrift_compact"} 2

No special metrics for the exporter.

The following can be used to check the current state of the connection to the remote Jaeger collector:

# HELP otelcol_jaegerexporter_conn_state Last connection state: 0 = Idle, 1 = Connecting, 2 = Ready, 3 = TransientFailure, 4 = Shutdown
# TYPE otelcol_jaegerexporter_conn_state gauge
otelcol_jaegerexporter_conn_state{exporter_name="jaeger",service_instance_id="..."} 2

[jpkrohling]

I'll leave this open for further feedback, but I think this is done for now.

[alolita]

@bogdandrutu can this be moved to a Phase 3 backlog which will be done post GA?

[alolita]

@jpkrohling what is the status on this issue? Is this done?

[jpkrohling]

I think we can close this for now. This is blocked, as we need to adopt otel-sdk before we can start adding the required metrics to the existing components in a sustainable manner.

[jpkrohling]

To be clear: I think we should still do it, but perhaps not for GA?

[alolita]

Move to post-GA backlog.

[tigrannajaryan]

@jpkrohling do we assume that once this issue is resolved we will also have a guidance on how custom metrics may be emitted by components? Like for example memory_limiter may emit a metric that describes whether it is currently limiting or no.

[jpkrohling]

I just created another issue to track that specific item: #4198.