[receiver/prometheus] Can a collector with prometheusreceiver receive data from a Prometheus server?

[mariana-mendes]

Component(s)

receiver/prometheus

Describe the issue you're reporting

Hello!
It is not clear to me if it's possible to use the following strategy to get metrics:

apps --> Prometheus server scraping metrics --> otel-collector with a prometheus receiver.
I've tried with the following configs:

my prometheus.yaml config:

...

    remoteWrite:
      - url:  https://<MY-OTEL-HOST-http>/v1/metrics


  # Prometheus scrape configs - start
  serverFiles:
    prometheus.yml:
      scrape_configs:
        - job_name: prometheus
          honor_labels: true
          honor_timestamps: true
          static_configs:
            - targets:
                - localhost:9090

...
        

my collector.yaml config:

istio:
  endpoints:
    - name: grpc-otlp
      port: 4327
      hosts:
        - <MY-OTEL-HOST-grpc>
      gateways:
        - istio-system/internal-gateway
    - name: otlp-http
      port: 4318
      hosts:
        - <MY-OTEL-HOST-http>
      gateways:
        - istio-system/internal-gateway


opentelemetry-collector:
...

  config:
    receivers:
      otlp:
        protocols:
          http:
            endpoint: 0.0.0.0:4318
      prometheus:
        config:
          scrape_configs:
            - job_name: prometheus
              honor_labels: true
              honor_timestamps: true
              static_configs:
              - targets:
                  - localhost:9090

    exporters:
      prometheusremotewrite:
        endpoint: http://<ENDPOINT-TO-MYDATA-SINK>/insert/0/prometheus/
 

      logging:
        verbosity: basic
        sampling_initial: 10
        sampling_thereafter: 1000

    processors:
...
    


...
      pipelines:
        logs: null
        metrics:
          receivers:
            - otlp
            - prometheus
          processors:
            - memory_limiter
            - batch
          exporters:
            - prometheusremotewrite
            - logging
        traces: null

The data is not reaching the collector, and in the Prometheus server logs, I'm getting the following error:

ts=2024-02-23T14:32:28.107Z caller=dedupe.go:112 component=remote level=error remote_name=4d846f url=http://<MY-OTEL-HOST-http>/v1/metrics msg=“non-recoverable error” count=11 exemplarCount=0 err=“server returned HTTP status 400 Bad Request: \u0008\u0003\u0012$unsupported Content-Encoding: snappy”

I'd like to know if I need to include something in the configuration or if it's really not possible.

Thank you!! 🤗

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

It looks like you are trying to write the prometheus remote write protocol to the OTLP receiver, which will not work since they are different protocols. The only way i'm aware of doing this today is to use the prometheus receiver and scrape the federated endpoint of the prometheus server. You will need to set honor_labels: true in the prometheus receiver config.

There is no prometheus remote write receiver at the moment.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

Feel free to reopen if you have further questions. The #otel-collector channel on slack is also a good place for questions like this