-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OTLP exporter not exporting ValueRecorder & ValueObserver to Prometheus backend #875
Comments
I believe this is a known issue addressed by open-telemetry/oteps#118. The ValueRecorder and ValueObserver events are becoming Summary values in Prometheus, and we will change this default behavior to Gauge. Can you look for a |
I just updated to the newest version of OTEL v0.7.0. Now in the logging exporter the captured metrics are shown correctly:
However, still in prometheus I only get to see the counter instruments output:
|
Also 2 issues appeared after updating to 0.7.0
|
I will investigate. Thank you for reporting this! |
Can you double-check that you're running the 0.7 release? I ask because the
|
I can confirm that as of my previous comment about the |
Would you state the OTel-Collector version that you tested with, just to help isolate this? |
I tested this using |
@zeyadkhaled I tried to reproduce your example, and got stuck at the |
You can find it in this readme
https://github.com/zeyadkhaled/openversion/blob/master/internal/sqlfiles/README.md
gomigrate was just an alias I created I think it goes under the name migrate
…On Wed, 1 Jul 2020 at 9:42 PM Joshua MacDonald ***@***.***> wrote:
@zeyadkhaled <https://github.com/zeyadkhaled> I tried to reproduce your
example, and got stuck at the gomigrate step. Where does this tool come
from?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#875 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AIPDEMMWGVWNBRBIPM7RTDLRZN7RXANCNFSM4OJJAI2A>
.
|
Thanks. I un-stuck myself on that point, and now managed to get everything running, but no metrics are arriving and I'm trying to diagnose the problem. (This is a good exercise for me, so I'll just keep at it.) |
I don't know if I should open a new issue for this or not. But I updated to the latest 0.5.0 version of collector and 0.7.0 of otel, and now in addition to the problem I created a branch for this issue to be able to reproduce it: https://github.com/zeyadkhaled/openversion/tree/upgrade-collector-and-otel-versions |
I don't think we need a new issue. There are now several well-known release coordination problems between the collector's OTLP protocol and the various language SDKs. I will work toward fixing this in the latest configuration, and thanks for the branch. |
One thing I'm aware of, in this code, is the call to I apologize, but the OTLP protocol is really unstable and one of the issues is that we haven't sorted out how to represent raw data (which is implied by Exact aggregation). I'm sure that doesn't explain everything going on here, but I wanted to pass it along while I investigate. |
I stumbled over a few more issues while debugging this, and I now think that the only necessary change in the client side is to switch to Then in the collector, somehow in the translation from OTLP into OpenCensus data, which is used by the Prometheus exporters, the ValueRecorder data is dropped and the Counter metadata becomes "gauge", not "counter". I will have to dig in on the collector side now. I do have a number of notes on the things that gave me trouble, I will batch those up and send when I'm able to get the end-to-end working. |
I've reported an issue in the collector to hopefully reduce the number of people who stumble into this, while we fix it. |
If you find an issue that might be a good first issue, I would more than love to help and contribute. |
I tried to upgrade the OTEL version on the master of my repo, I made a branch for this issue: |
@zeyadkhaled Yes, you've discovered a duplicate of #887. I am investigating this. |
This should be fixed in the 0.8 release. |
However note that the OTLP receiver is still not working in this case (open-telemetry/opentelemetry-collector#1255). |
Describe the bug
I am using an OTLP exporter and metrics pusher with otel-collector with a Prometheus exporter and Logging exporter.
In the Logging exporter, all collected metrics are showing but in Prometheus backend, only ValueCounter instrument values are showing.
I am using the latest otel-collector-contrib image and running it alongside a demo service, some DBs, latest Prometheus image using docker.
What config did you use?
Environment
OS: MacOS Catalina
Compiler(if manually compiled): go 1.14
Additional context
The code I wrote using OTEL format to export metrics works because I added a Prometheus receiver in the collector and a Prometheus exporter without changing any of the counter and value recorders code and it worked fine. So the issue is most likely associated with OTEL-exporter not able to export some Metric values that prometheus backend can identify.
I have this code with a demo service in the repo:
https://github.com/zeyadkhaled/OpenTelemetry-Go-Project-with-Collector-and-OTLP-Exporter
The text was updated successfully, but these errors were encountered: