Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rpc.server.duration metric does not have a description #4301

Closed
puckpuck opened this issue Sep 15, 2023 · 0 comments · Fixed by #4302
Closed

rpc.server.duration metric does not have a description #4301

puckpuck opened this issue Sep 15, 2023 · 0 comments · Fixed by #4302
Labels
area: instrumentation Related to an instrumentation package bug Something isn't working instrumentation: otelgrpc

Comments

@puckpuck
Copy link
Contributor

Description

A typical metrics backend is Prometheus. When used with OpenTelemetry SDKs, the suggested deployment is to route telemetry data through an OpenTelemetry Collector configured with a Prometheus exporter. In this configuration, when other language SDKs (ie: Java) are part of the request path, all metrics must have the same description. Metrics that match on name, but not on description will be dropped by the Prometheus exporter from previously received metrics.

We have run into this issue on the OpenTelemetry Demo project, our workaround is to write a special transform rule to remove descriptions from offending metrics.

The is the error that we see in the OpenTelemetry collector logs:

2023-09-15T22:55:59.324Z	error	prometheusexporter@v0.84.0/log.go:23	error gathering metrics: collected metric rpc_server_duration label:{name:"container_id"  value:"988d641de9f1cc2813edfc8e859fb2dc6a198f9c8d10d25f5297e3e96fedf488"}  label:{name:"host_arch"  value:"aarch64"}  label:{name:"host_name"  value:"988d641de9f1"}  label:{name:"job"  value:"opentelemetry-demo/adservice"}  label:{name:"net_host_name"  value:"adservice"}  label:{name:"os_description"  value:"Linux 6.3.13-linuxkit"}  label:{name:"os_type"  value:"linux"}  label:{name:"process_command_line"  value:"/opt/java/openjdk/bin/java -javaagent:/usr/src/app/opentelemetry-javaagent.jar oteldemo.AdService"}  label:{name:"process_executable_path"  value:"/opt/java/openjdk/bin/java"}  label:{name:"process_pid"  value:"1"}  label:{name:"process_runtime_description"  value:"Eclipse Adoptium OpenJDK 64-Bit Server VM 17.0.8.1+1"}  label:{name:"process_runtime_name"  value:"OpenJDK Runtime Environment"}  label:{name:"process_runtime_version"  value:"17.0.8.1+1"}  label:{name:"rpc_grpc_status_code"  value:"0"}  label:{name:"rpc_method"  value:"GetAds"}  label:{name:"rpc_service"  value:"oteldemo.AdService"}  label:{name:"rpc_system"  value:"grpc"}  label:{name:"service_name"  value:"adservice"}  label:{name:"service_namespace"  value:"opentelemetry-demo"}  label:{name:"telemetry_auto_version"  value:"1.28.0"}  label:{name:"telemetry_sdk_language"  value:"java"}  label:{name:"telemetry_sdk_name"  value:"opentelemetry"}  label:{name:"telemetry_sdk_version"  value:"1.28.0"}  histogram:{sample_count:269  sample_sum:884.9096299999994  bucket:{cumulative_count:0  upper_bound:0}  bucket:{cumulative_count:243  upper_bound:5}  bucket:{cumulative_count:263  upper_bound:10}  bucket:{cumulative_count:267  upper_bound:25}  bucket:{cumulative_count:268  upper_bound:50}  bucket:{cumulative_count:268  upper_bound:75}  bucket:{cumulative_count:269  upper_bound:100}  bucket:{cumulative_count:269  upper_bound:250}  bucket:{cumulative_count:269  upper_bound:500}  bucket:{cumulative_count:269  upper_bound:750}  bucket:{cumulative_count:269  upper_bound:1000}  bucket:{cumulative_count:269  upper_bound:2500}  bucket:{cumulative_count:269  upper_bound:5000}  bucket:{cumulative_count:269  upper_bound:7500}  bucket:{cumulative_count:269  upper_bound:10000}} has help "The duration of an inbound RPC invocation" but should have ""
	{"kind": "exporter", "data_type": "metrics", "name": "prometheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*promLogger).Println
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.84.0/log.go:23
github.com/prometheus/client_golang/prometheus/promhttp.HandlerForTransactional.func1
	github.com/prometheus/client_golang@v1.16.0/prometheus/promhttp/http.go:144
net/http.HandlerFunc.ServeHTTP
	net/http/server.go:2136
net/http.(*ServeMux).ServeHTTP
	net/http/server.go:2514
go.opentelemetry.io/collector/config/confighttp.(*decompressor).ServeHTTP
	go.opentelemetry.io/collector/config/confighttp@v0.84.0/compression.go:147
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Handler).ServeHTTP
	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.42.0/handler.go:212
go.opentelemetry.io/collector/config/confighttp.(*clientInfoHandler).ServeHTTP
	go.opentelemetry.io/collector/config/confighttp@v0.84.0/clientinfohandler.go:28
net/http.serverHandler.ServeHTTP
	net/http/server.go:2938
net/http.(*conn).serve
	net/http/server.go:2009

Environment

  • OS: Linux
  • Architecture: arm
  • Go Version: 1.19
  • otelgrpc version: v0.43.0

Steps To Reproduce

Use the gRPC instrumentation library for a gRPC service, with a MeterProvider configured

	var srv = grpc.NewServer(
		grpc.UnaryInterceptor(otelgrpc.UnaryServerInterceptor()),
		grpc.StreamInterceptor(otelgrpc.StreamServerInterceptor()),
	)

The description for the rpc.sever.duration metric will be blank.

Expected behavior

The description for the rpc.server.duration metric should be: Measures the duration of inbound RPC.

This is defined in the OpenTelemetry semantic convention for this metric.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: instrumentation Related to an instrumentation package bug Something isn't working instrumentation: otelgrpc
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants