Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[exporter/prometheus] Allow setting custom "job" and "instance" attributes #33118

Closed
swar8080 opened this issue May 18, 2024 · 3 comments
Closed
Labels
bug Something isn't working closed as inactive exporter/prometheus needs triage New item requiring triage Stale

Comments

@swar8080
Copy link
Contributor

swar8080 commented May 18, 2024

Component(s)

exporter/prometheus

What happened?

Description

prometheusexporter automatically sets the job and instance attributes on metrics exposed for scraping, which are derived from the metric's resource attributes.

If you try setting custom job/instance values then the conversion to prometheus metric data model fails because of duplicate labels:

Steps to Reproduce

Submit a metric to this component with either of these attributes set. Then call its /metrics endpoint

Expected Result

Custom label values take priority over the default values

Actual Result

Setting custom label values breaks scraping for these metrics

Collector version

v0.100.0

Environment information

Reproduced locally but this should be reproducible on all environments

OpenTelemetry Collector configuration

exporters:
  prometheus:
    endpoint: "0.0.0.0:9465"
    enable_open_metrics: true
    metric_expiration: 99m
    send_timestamps: false

  transform:
    error_mode: ignore
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["job"], "custom")

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [transform]
      exporters: [prometheus]

Log output

2024-05-17T18:45:38.607-0400    error   prometheusexporter@v0.100.0/collector.go:386    failed to convert metric calls: duplicate label names in constant and variable labels for metric "calls_total"        {"service": "my-instance", "kind": "exporter", "data_type": "metrics", "name": "prometheus/exporter"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
        /Users/sswartz/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.100.0/collector.go:386
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        /Users/sswartz/go/pkg/mod/github.com/prometheus/client_golang@v1.19.1/prometheus/registry.go:457

Additional context

Our use case is migrating some metrics to OpenTelemetry collector, so we want to preserve the existing job values to avoid breaking existing dashboards/alerts

I've patched this in our custom fork by checking if job/instance are already set before setting the default

@swar8080 swar8080 added bug Something isn't working needs triage New item requiring triage labels May 18, 2024
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

Copy link
Contributor

This issue has been closed as inactive because it has been stale for 120 days with no activity.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Sep 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working closed as inactive exporter/prometheus needs triage New item requiring triage Stale
Projects
None yet
Development

No branches or pull requests

1 participant