influxdbexporter does not support empty tag values from hostmetrics process scraper

[padraic-padraic]

Component(s)

exporter/influxdb, receiver/hostmetrics

What happened?

Description

For v0.75.0 and up, I can no-longer combine the process scraper from the hostmetrics receiver with the influxdb exporter.

Trying to do so results in the following error message when converting from OTLP to InfluxDB Line Protocol, whether I use telegraf-prometheus-v1 or telegraf-prometheus-v2 as the metrics_schema:

otel_contrib | 2023-05-03T11:53:17.416Z error exporterhelper/queued_retry.go:401 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "influxdb", "error": "Permanent error: failed to convert OTLP metric to line protocol: failed to write point for sum: Permanent error: failed to encode point: encoding point 68: invalid tag value process.command=\"\"", "dropped_items": 1269}

As mentioned in other issues (e.g. #20435), sometimes the process scraper is not able to collect all the attributes of a given process. However, rather than dropping the affected series, the error causes the influxdbexporter to drop all the collected data.

I've attempted a workaround using the filter and attributes procesor, but haven't been able to get something working.

The issue occurs only on v.075.0 and up: the attached config will work for 0.74.0. Could the issue be related to the changes made to pkg/ottl or to the processscraper in the 0.75.0 release?

Steps to Reproduce

Run the otel-collector with the attached config

Expected Result

Metrics are exported to InfluxDB

Actual Result

Data is dropped as some metrics cannot be converted to InfluxDB line protocol.

Collector version

0.75.0

Environment information

Environment

Tested with:

• OS: Ubuntu 18.04.6 LTS
  Arch: amd64
  Version: 0.74.0 and 0.76.1 (DEB)
• Docker: Tags 0.74.0, 0.75.0 and 0.76.1
  Arch: amd64

OpenTelemetry Collector configuration

receivers:
  hostmetrics:
    scrapers:
      cpu:
      memory:
      filesystem:
      paging:
      process:
        metrics:
          process.cpu.time:
            enabled: false
          process.memory.utilization:
            enabled: true


processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 10
    spike_limit_percentage: 7
  batch:
    timeout: 10s

exporters:
  influxdb:
    endpoint: https:/localhost:8086
    bucket: "example_bucket"
    token: "writer:writer_password"
    org: "dummy_val"

service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [memory_limiter, batch]
      exporters: [influxdb]

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/influxdb: @jacobmarble
• receiver/hostmetrics: @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[wise0wl]

This is a major problem and essentially makes the InfluxDB exporter unusable.

[jacobmarble]

Thanks for the example config, that should help me reproduce the issue.

[jacobmarble]

Please assign this issue to me.

[atoulme]

Done, it's yours!

[jacobmarble]

Since otelcol-contrib v0.75.0, the hostmetrics receiver is emitting metrics with empty string attribute values, such as device = string(""). I've confirmed that this behavior started in v0.75.0.

The InfluxDB line protocol encoder has rejected empty tag values for at least three years.

Nevertheless, the InfluxDB exporter can do better by silently dropping these attributes (which is semantically equivalent in InfluxDB). I'll prepare a PR.

[atoulme]

Would you mind opening a bug for the hostmetricsreceiver to look after empty attribute values?

[padraic-padraic]

Thank you very much all for the quick response!