Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[translator/prometheusremotewrite] add metric name to error message #18292

Conversation

jmichalek132
Copy link
Contributor

@jmichalek132 jmichalek132 commented Feb 5, 2023

Description:

Right now when using Prometheus remote write exporter some metrics that are not supported by the exporter are dropped:

⚠️ Non-cumulative monotonic, histogram, and summary OTLP metrics are dropped by this exporter.

At the moment, it gets logged and it also propagates into metrics but there is no way to tell which metrics are affected.
Because of that, I would like to suggest adding the metric name to the error message which gets logged so we can tell which metrics are being dropped.

Link to tracking Issue:

Testing:
Tested in our testing environment with locally built image, example log line:

opentelemetry-cluster-5f9cc5d595-pbcf8 opentelemetry-collector 2023-02-06T13:59:01.448Z	error	exporterhelper/queued_retry.go:394	Exporting failed. The error is not retryable. Dropping data.	{"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: invalid temporality and type combination. system.network.dropped is dropped; invalid temporality and type combination. system.network.io is dropped; invalid temporality and type combination. system.network.packets is dropped", "dropped_items": 200}

Documentation:

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Feb 5, 2023

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: jmichalek132 / name: Juraj Michálek (9dd6d9f)

@jmichalek132 jmichalek132 marked this pull request as ready for review February 6, 2023 14:15
@jmichalek132 jmichalek132 requested a review from a team February 6, 2023 14:15
@kovrus
Copy link
Member

kovrus commented Feb 7, 2023

@jmichalek132
Copy link
Contributor Author

jmichalek132 commented Feb 8, 2023

@runforesight
Copy link

runforesight bot commented Feb 8, 2023

Foresight Summary

    
Major Impacts

build-and-test duration(8 minutes 50 seconds) has decreased 57 minutes 11 seconds compared to main branch avg(1 hour 6 minutes 1 second).
View More Details

⭕  build-and-test-windows workflow has finished in 5 seconds (40 minutes 55 seconds less than main branch avg.) and finished at 15th Feb, 2023.


Job Failed Steps Tests
windows-unittest-matrix -     🔗  N/A See Details
windows-unittest -     🔗  N/A See Details

✅  check-links workflow has finished in 43 seconds (1 minute 50 seconds less than main branch avg.) and finished at 15th Feb, 2023.


Job Failed Steps Tests
changed files -     🔗  N/A See Details
check-links -     🔗  N/A See Details

✅  telemetrygen workflow has finished in 1 minute (2 minutes 5 seconds less than main branch avg.) and finished at 15th Feb, 2023.


Job Failed Steps Tests
build-dev -     🔗  N/A See Details
publish-latest -     🔗  N/A See Details
publish-stable -     🔗  N/A See Details

✅  changelog workflow has finished in 1 minute 37 seconds (2 minutes 6 seconds less than main branch avg.) and finished at 15th Feb, 2023.


Job Failed Steps Tests
changelog -     🔗  N/A See Details

✅  prometheus-compliance-tests workflow has finished in 4 minutes 3 seconds (4 minutes 50 seconds less than main branch avg.) and finished at 15th Feb, 2023.


Job Failed Steps Tests
prometheus-compliance-tests -     🔗  ✅ 21  ❌ 0  ⏭ 0    🔗 See Details

✅  load-tests workflow has finished in 10 minutes 22 seconds (6 minutes 35 seconds less than main branch avg.) and finished at 15th Feb, 2023.


Job Failed Steps Tests
loadtest (TestIdleMode) -     🔗  ✅ 1  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestTraceAttributesProcessor) -     🔗  ✅ 3  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestMetric10kDPS|TestMetricsFromFile) -     🔗  ✅ 6  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestMetricResourceProcessor|TestTrace10kSPS) -     🔗  ✅ 12  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestTraceBallast1kSPSWithAttrs|TestTraceBallast1kSPSAddAttrs) -     🔗  ✅ 10  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestTraceNoBackend10kSPS|TestTrace1kSPSWithAttrs) -     🔗  ✅ 8  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestBallastMemory|TestLog10kDPS) -     🔗  ✅ 18  ❌ 0  ⏭ 0    🔗 See Details
setup-environment -     🔗  N/A See Details

✅  e2e-tests workflow has finished in 11 minutes 24 seconds (5 minutes 1 second less than main branch avg.) and finished at 15th Feb, 2023.


Job Failed Steps Tests
kubernetes-test -     🔗  N/A See Details

✅  build-and-test workflow has finished in 8 minutes 50 seconds (57 minutes 11 seconds less than main branch avg.) and finished at 16th Feb, 2023.


Job Failed Steps Tests
build-package (deb) -     🔗  N/A See Details
publish-check -     🔗  N/A See Details
publish-dev -     🔗  N/A See Details
publish-stable -     🔗  N/A See Details

🔎 See details on Foresight

*You can configure Foresight comments in your organization settings page.

@atoulme atoulme added the ready to merge Code review completed; ready to merge by maintainers label Feb 15, 2023
Co-authored-by: Anthony Mirabella <a9@aneurysm9.com>
@codeboten codeboten merged commit 30cf583 into open-telemetry:main Feb 16, 2023
@jmichalek132 jmichalek132 deleted the jm-add-metric-name-to-error-message-when-invalid branch February 18, 2023 12:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pkg/translator/prometheus ready to merge Code review completed; ready to merge by maintainers
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants