OTLP exporter not exporting ValueRecorder & ValueObserver to Prometheus backend #875

zeyadkhaled · 2020-06-26T12:25:27Z

Describe the bug
I am using an OTLP exporter and metrics pusher with otel-collector with a Prometheus exporter and Logging exporter.
In the Logging exporter, all collected metrics are showing but in Prometheus backend, only ValueCounter instrument values are showing.

I am using the latest otel-collector-contrib image and running it alongside a demo service, some DBs, latest Prometheus image using docker.

What config did you use?

otel-collector-config.yaml

receivers:
  otlp:
    endpoint: 0.0.0.0:55678

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    namespace: versionsvc

  logging:
    loglevel: debug
    
  stackdriver:
    project: digital-waters-276111
    metric_prefix: versionsvc
    number_of_workers: 3
    skip_create_metric_descriptor: true

processors:
  batch:
  queued_retry:

extensions:
  health_check:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging,stackdriver]
      processors: [batch, queued_retry]
    metrics:
      receivers: [otlp]
      exporters: [logging,prometheus]

creation of the exporter and pusher:


func initProviders() (*otlp.Exporter, *push.Controller) {
	collectorAddr, ok := os.LookupEnv("OTEL_RECIEVER_ENDPOINT")
	if !ok {
		collectorAddr = otlp.DefaultCollectorHost + ":" + string(otlp.DefaultCollectorHost)
	}
	exporter, err := otlp.NewExporter(otlp.WithAddress(collectorAddr),
		otlp.WithInsecure(),
		otlp.WithGRPCDialOption(grpc.WithBlock()))

	if err != nil {
		log.Fatal(err)
	}

	tp, err := sdktrace.NewProvider(
		sdktrace.WithConfig(sdktrace.Config{DefaultSampler: sdktrace.AlwaysSample()}),
		sdktrace.WithSyncer(exporter))
	if err != nil {
		log.Fatal(err)
	}

	global.SetTraceProvider(tp)

	pusher := push.New(
		simple.NewWithExactDistribution(),
		exporter,
		push.WithStateful(true),
		push.WithPeriod(2*time.Second),
	)

	global.SetMeterProvider(pusher.Provider())
	pusher.Start()
	return exporter, pusher
}

docker-compose.yaml

version: "3.1"
services:

  redis:
    image: redis:4
    ports:
      - "6379:6379"
    entrypoint: 
      "redis-server"

  db:
    image: postgres:11
    ports:
      - "5432:5432"
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: roottoor
      POSTGRES_DB: backend

  open-telemetry-demo:
    build: ../.
    environment:
      - GO111MODULE=on
      - OTEL_RECIEVER_ENDPOINT=otel-collector:55678
    depends_on:
      - otel-collector
      - db 
      - redis   
    ports: 
      - "8088:8088"

  otel-collector:
    image: ${OTELCOL_IMG}
    command: ["--config=/etc/otel-collector-config.yaml", "${OTELCOL_ARGS}"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "1888:1888"   
      - "8888:8888"   
      - "8889:8889"   
      - "13133:13133" 
      - "55678:55678"       
      - "55680:55679"
  
  prometheus:
    container_name: prometheus
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

prometheus.yaml

scrape_configs:
    - job_name: 'otel-collector'
      scrape_interval: 1s
      static_configs:
        - targets: ['otel-collector:8889']

Environment
OS: MacOS Catalina
Compiler(if manually compiled): go 1.14

Additional context

The code I wrote using OTEL format to export metrics works because I added a Prometheus receiver in the collector and a Prometheus exporter without changing any of the counter and value recorders code and it worked fine. So the issue is most likely associated with OTEL-exporter not able to export some Metric values that prometheus backend can identify.
I have this code with a demo service in the repo:

https://github.com/zeyadkhaled/OpenTelemetry-Go-Project-with-Collector-and-OTLP-Exporter

The text was updated successfully, but these errors were encountered:

jmacd · 2020-06-29T15:43:11Z

I believe this is a known issue addressed by open-telemetry/oteps#118. The ValueRecorder and ValueObserver events are becoming Summary values in Prometheus, and we will change this default behavior to Gauge. Can you look for a _sum metric w/ the same name prefix?

zeyadkhaled · 2020-06-30T10:48:29Z

I just updated to the newest version of OTEL v0.7.0. Now in the logging exporter the captured metrics are shown correctly:

InstrumentationLibrary service.meter 
otel-collector_1       | Metric #0
otel-collector_1       | Descriptor:
otel-collector_1       |      -> Name: api.hit.count
otel-collector_1       |      -> Description: 
otel-collector_1       |      -> Unit: 
otel-collector_1       |      -> Type: INT64
otel-collector_1       | Metric #1
otel-collector_1       | Descriptor:
otel-collector_1       |      -> Name: bytes.recieved
otel-collector_1       |      -> Description: 
otel-collector_1       |      -> Unit: 
otel-collector_1       |      -> Type: SUMMARY
otel-collector_1       | Metric #2
otel-collector_1       | Descriptor:
otel-collector_1       |      -> Name: errors.counter
otel-collector_1       |      -> Description: 
otel-collector_1       |      -> Unit: 
otel-collector_1       |      -> Type: INT64
otel-collector_1       | Metric #3
otel-collector_1       | Descriptor:
otel-collector_1       |      -> Name: process.duration
otel-collector_1       |      -> Description: 
otel-collector_1       |      -> Unit: 
otel-collector_1       |      -> Type: SUMMARY

However, still in prometheus I only get to see the counter instruments output:

# HELP versionsvc_api_hit_count 
# TYPE versionsvc_api_hit_count gauge
versionsvc_api_hit_count{endpoint="api.endpoint:/v2/version:GET"} 1
# HELP versionsvc_errors_counter 
# TYPE versionsvc_errors_counter gauge
versionsvc_errors_counter{service_method="Get"} 1

zeyadkhaled · 2020-06-30T10:52:37Z

Also 2 issues appeared after updating to 0.7.0

Suddenly the service that initialized the exporters starts outputting:

open-telemetry-demo_1  | 2020/06/30 10:39:04 exporter disconnected
open-telemetry-demo_1  | 2020/06/30 10:39:06 exporter disconnected
open-telemetry-demo_1  | 2020/06/30 10:39:08 exporter disconnected
open-telemetry-demo_1  | 2020/06/30 10:39:10 exporter disconnected
open-telemetry-demo_1  | 2020/06/30 10:39:12 exporter disconnected

I think this is related with the first issue. For counter values, only the latest value is captured in Prometheus and the old ones are erased. What I mean is for the same instrument versionsvc_api_hit_count I only ever get the value 1.

jmacd · 2020-06-30T15:12:28Z

I will investigate. Thank you for reporting this!

jmacd · 2020-06-30T15:52:09Z

Can you double-check that you're running the 0.7 release? I ask because the push.Stateful() method has disappeared in 0.7. I suspect the "exporter disconnected" bit is related to the breaking OTLP change that hit us in this release cycle.

	pusher := push.New(
		simple.NewWithExactDistribution(),
		exporter,
		push.WithStateful(true),
		push.WithPeriod(2*time.Second),
	)

zeyadkhaled · 2020-06-30T18:18:25Z

I can confirm that as of my previous comment about the exporter disconnected I am using 0.7 release and I did need to remove push.WithStateful() while writing the previous comment.

jmacd · 2020-06-30T22:15:41Z

Would you state the OTel-Collector version that you tested with, just to help isolate this?

zeyadkhaled · 2020-07-01T06:31:37Z

I tested this using otel/opentelemetry-collector:latest which happens to be v0.4.0

jmacd · 2020-07-01T18:42:21Z

@zeyadkhaled I tried to reproduce your example, and got stuck at the gomigrate step. Where does this tool come from?

zeyadkhaled · 2020-07-01T20:14:44Z

You can find it in this readme https://github.com/zeyadkhaled/openversion/blob/master/internal/sqlfiles/README.md gomigrate was just an alias I created I think it goes under the name migrate

…

On Wed, 1 Jul 2020 at 9:42 PM Joshua MacDonald ***@***.***> wrote: @zeyadkhaled <https://github.com/zeyadkhaled> I tried to reproduce your example, and got stuck at the gomigrate step. Where does this tool come from? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#875 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIPDEMMWGVWNBRBIPM7RTDLRZN7RXANCNFSM4OJJAI2A> .

jmacd · 2020-07-01T23:03:39Z

Thanks. I un-stuck myself on that point, and now managed to get everything running, but no metrics are arriving and I'm trying to diagnose the problem. (This is a good exercise for me, so I'll just keep at it.)

zeyadkhaled · 2020-07-02T08:28:41Z

I don't know if I should open a new issue for this or not. But I updated to the latest 0.5.0 version of collector and 0.7.0 of otel, and now in addition to the problem "Recorder and Observer values are missing" now the counter metrics are being overwritten with every new incoming value.

I created a branch for this issue to be able to reproduce it:

https://github.com/zeyadkhaled/openversion/tree/upgrade-collector-and-otel-versions

jmacd · 2020-07-02T15:46:43Z

I don't think we need a new issue. There are now several well-known release coordination problems between the collector's OTLP protocol and the various language SDKs. I will work toward fixing this in the latest configuration, and thanks for the branch.

jmacd · 2020-07-02T16:03:31Z

One thing I'm aware of, in this code, is the call to simple.NewWithExactDistribution() has to be replaced simple.NewWithInexpensiveDistribution() for the OTLP metric exporter. The OTLP exporter supports only Sum and MinMaxSumCount aggregation in the current release.

I apologize, but the OTLP protocol is really unstable and one of the issues is that we haven't sorted out how to represent raw data (which is implied by Exact aggregation). I'm sure that doesn't explain everything going on here, but I wanted to pass it along while I investigate.

jmacd · 2020-07-02T19:10:17Z

I stumbled over a few more issues while debugging this, and I now think that the only necessary change in the client side is to switch to simple.NewWithInexpensiveDistribution. This encodes a SummaryDataPoint in OTLP which I see printed in the logging exporter on the collector side. (Related improvements: see open-telemetry/oteps#117 and open-telemetry/oteps#118).

Then in the collector, somehow in the translation from OTLP into OpenCensus data, which is used by the Prometheus exporters, the ValueRecorder data is dropped and the Counter metadata becomes "gauge", not "counter". I will have to dig in on the collector side now.

I do have a number of notes on the things that gave me trouble, I will batch those up and send when I'm able to get the end-to-end working.

jmacd · 2020-07-02T19:17:30Z

I've reported an issue in the collector to hopefully reduce the number of people who stumble into this, while we fix it.

open-telemetry/opentelemetry-collector#1255

zeyadkhaled · 2020-07-02T20:10:19Z

If you find an issue that might be a good first issue, I would more than love to help and contribute.

zeyadkhaled · 2020-07-03T07:13:57Z

I tried to upgrade the OTEL version on the master of my repo, which uses for the Metrics pipeline a Prometheus receiver and Prometheus exporter and in the code uses a Prom Exporter and the OTLP exporter is just used for traces, and realized after sending a request the counter of api.hit for example starts from "0" not from 1 and starts doubling every 5 seconds almost and it infinitely keeps increasing.

I made a branch for this issue:
https://github.com/zeyadkhaled/openversion/tree/prom-exporter-otel0.7.0-issue

jmacd · 2020-07-03T16:14:14Z

@zeyadkhaled Yes, you've discovered a duplicate of #887. I am investigating this.

jmacd · 2020-07-16T22:00:06Z

This should be fixed in the 0.8 release.

jmacd · 2020-07-16T22:00:26Z

However note that the OTLP receiver is still not working in this case (open-telemetry/opentelemetry-collector#1255).

jmacd added the area:metrics Part of OpenTelemetry Metrics label Jun 29, 2020

jmacd self-assigned this Jun 30, 2020

jmacd mentioned this issue Jul 2, 2020

Use direct API calls; allocate instruments once zeyadkhaled/openversion#1

Merged

jmacd mentioned this issue Jul 2, 2020

Issues using metrics + OTEL collector + prometheus exporter #886

Closed

jmacd mentioned this issue Jul 2, 2020

Prometheus export from OTLP receiver not functional open-telemetry/opentelemetry-collector#1255

Closed

MrAlias added area: exporter bug Something isn't working labels Jul 2, 2020

jmacd mentioned this issue Jul 2, 2020

Metrics aggregation for same metric with 2 or more nodes open-telemetry/opentelemetry-collector#433

Closed

jmacd closed this as completed Jul 16, 2020

jmacd reopened this Jul 16, 2020

jmacd closed this as completed Jul 16, 2020

jmacd mentioned this issue Jul 17, 2020

Add Last value to MinMaxSumCount OTLP data point a.k.a. "MMLSC" open-telemetry/oteps#117

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OTLP exporter not exporting ValueRecorder & ValueObserver to Prometheus backend #875

OTLP exporter not exporting ValueRecorder & ValueObserver to Prometheus backend #875

zeyadkhaled commented Jun 26, 2020

jmacd commented Jun 29, 2020

zeyadkhaled commented Jun 30, 2020

zeyadkhaled commented Jun 30, 2020

jmacd commented Jun 30, 2020

jmacd commented Jun 30, 2020

zeyadkhaled commented Jun 30, 2020

jmacd commented Jun 30, 2020

zeyadkhaled commented Jul 1, 2020

jmacd commented Jul 1, 2020

zeyadkhaled commented Jul 1, 2020 via email

jmacd commented Jul 1, 2020

zeyadkhaled commented Jul 2, 2020

jmacd commented Jul 2, 2020

jmacd commented Jul 2, 2020

jmacd commented Jul 2, 2020

jmacd commented Jul 2, 2020

zeyadkhaled commented Jul 2, 2020

zeyadkhaled commented Jul 3, 2020 •

edited

Loading

jmacd commented Jul 3, 2020

jmacd commented Jul 16, 2020

jmacd commented Jul 16, 2020

OTLP exporter not exporting ValueRecorder & ValueObserver to Prometheus backend #875

OTLP exporter not exporting ValueRecorder & ValueObserver to Prometheus backend #875

Comments

zeyadkhaled commented Jun 26, 2020

jmacd commented Jun 29, 2020

zeyadkhaled commented Jun 30, 2020

zeyadkhaled commented Jun 30, 2020

jmacd commented Jun 30, 2020

jmacd commented Jun 30, 2020

zeyadkhaled commented Jun 30, 2020

jmacd commented Jun 30, 2020

zeyadkhaled commented Jul 1, 2020

jmacd commented Jul 1, 2020

zeyadkhaled commented Jul 1, 2020 via email

jmacd commented Jul 1, 2020

zeyadkhaled commented Jul 2, 2020

jmacd commented Jul 2, 2020

jmacd commented Jul 2, 2020

jmacd commented Jul 2, 2020

jmacd commented Jul 2, 2020

zeyadkhaled commented Jul 2, 2020

zeyadkhaled commented Jul 3, 2020 • edited Loading

jmacd commented Jul 3, 2020

jmacd commented Jul 16, 2020

jmacd commented Jul 16, 2020

zeyadkhaled commented Jul 3, 2020 •

edited

Loading