-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[exporter/prometheusremotewrite] Collector crashes while Scraping Histogram Metrics #12777
Comments
Pinging code owners: @Aneurysm9. See Adding Labels via Comments if you do not have permissions to add labels yourself. |
Pinging @dashpole @bertysentry as well as owners of the translator |
This is the offending code in function addSingleHistogramDataPoint() in // add le=+Inf bucket
infBucket := &prompb.Sample{
Timestamp: time,
}
if pt.Flags().HasFlag(pmetric.MetricDataPointFlagNoRecordedValue) {
infBucket.Value = math.Float64frombits(value.StaleNaN)
} else {
cumulativeCount += pt.BucketCounts().At(pt.BucketCounts().Len() - 1) // panic: runtime error: index out of range [-1]
infBucket.Value = float64(cumulativeCount)
} This part of the code is out of my scope of expertise, I'm sorry! @Aneurysm9 @dmitryax Any idea? |
Looks like it can panic on histograms without any buckets |
Fix: #12854 |
We were running into open-telemetry/opentelemetry-collector-contrib#12777 This was fixed by open-telemetry/opentelemetry-collector-contrib#12854 which was released in v0.57.2 which this upgrades to. Ideally #3852 should be merged but its a big PR, hence also sending this to make sure merging is quicker. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
We were running into open-telemetry/opentelemetry-collector-contrib#12777 This was fixed by open-telemetry/opentelemetry-collector-contrib#12854 which was released in v0.57.2 which this upgrades to.
We were running into open-telemetry/opentelemetry-collector-contrib#12777 This was fixed by open-telemetry/opentelemetry-collector-contrib#12854 which was released in v0.57.2 which this upgrades to. (cherry picked from commit bed543c)
We were running into open-telemetry/opentelemetry-collector-contrib#12777 This was fixed by open-telemetry/opentelemetry-collector-contrib#12854 which was released in v0.57.2 which this upgrades to. (cherry picked from commit bed543c)
We were running into open-telemetry/opentelemetry-collector-contrib#12777 This was fixed by open-telemetry/opentelemetry-collector-contrib#12854 which was released in v0.57.2 which this upgrades to. (cherry picked from commit bed543c) Co-authored-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Describe the bug
I'm running Opentelemetry collector contrib v0.56.0 Windows Amd64 binary with OTLP Receiver in Windows machine and have a client application in Spring boot v2.6.9 using Micrometer Registry OTLP v1.9.2 to publish metrics to collector binary in OTLP Format. But as soon as client application starts publishing the metrics, the collector Crashes.
Related to #7149
Steps to reproduce
OTLPMeterRegistry
Bean to start exposing the metrics in OTLP format and start the application.(The collector publish endpoint will be default i.e; http://localhost:4318/v1/metrics)What did you expect to see?
Metrics to be remote written to the specified remote write endpoint.
What did you see instead?
Collector crashes with below exception
What version did you use?
Version: 0.56.0
What config did you use?
Config:
Environment
OS: Windows
The text was updated successfully, but these errors were encountered: