noisy debug logs with compare string to non-string

[trajano]

Component(s)

processor/filter

What happened?

Description

Seeing a lot of debug noise, but does not show any reason how to fix it (note this may be due to compare.go not providing a facility to mute invalid comparisons.

Steps to Reproduce

Define the filter

Expected Result

Quiet

Actual Result

Lots of debug statements

Collector version

0.88.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

processors:
  filter/drop_actuator:
    traces:
      span:
        - (attributes["http.method"] != nil) and (attributes["http.method"] == "GET") and (attributes["http.route"] == "/actuator/prometheus")
service:
  telemetry:
    logs:
      level: debug
      encoding: json
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter/drop_actuator,batch]

Log output

{"level":"debug","ts":1699397830.919674,"caller":"ottl@v0.88.0/compare.go:22","msg":"string to non-string","kind":"processor","name":"filter/drop_actuator","pipeline":"traces","op":0}
{"level":"debug","ts":1699397832.4553528,"caller":"ottl@v0.88.0/compare.go:22","msg":"string to non-string","kind":"processor","name":"filter/drop_actuator","pipeline":"traces","op":1}
{"level":"debug","ts":1699397832.4554641,"caller":"ottl@v0.88.0/compare.go:22","msg":"string to non-string","kind":"processor","name":"filter/drop_actuator","pipeline":"traces","op":0}
{"level":"debug","ts":1699397832.4554925,"caller":"ottl@v0.88.0/compare.go:22","msg":"string to non-string","kind":"processor","name":"filter/drop_actuator","pipeline":"traces","op":1}

Additional context

I also tried

    error_mode: ignore

still noisy.

[github-actions]

Pinging code owners:

• processor/filter: @TylerHelmuth @boostchicken

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Hello @trajano, just to clarify, did you file this issue because you don't want so many logs in your collector's output, or because you want the specific message you're seeing to provide some kind of guidance to resolve the issue?

[trajano]

Too many logs in the collectors output

There's also two problems with the messaging

1. it does not provide a way of fixing the problem if there is one.
2. it does not tell me what specific data and ideally what line as well caused the problem (that could likely be done on compare.go

But I think the fix would be filter should do a cursory check or maybe compare can provide a validator before sending it to compare for processing.

[crobert-1]

I think the best option is probably to add more information to the log output, specifically which string and non-string values are being compared.

I don't think anything should be done with the filter processor here, or other logic changes. The check will fail either way and this log message is really showing that we're properly validating data in our checks, and that everything is working as intended. The logic that's being used to compare values is in the ottl package which is widely used across different collector components. Because of this, I don't think it's a good idea to start adding logic around it in different components.

The log message is also showing where the check is coming from (filter/drop_actuator). From this, we know the comparisons defined here in your processor are what's logging errors:

        - (attributes["http.method"] != nil) and (attributes["http.method"] == "GET") and (attributes["http.route"] == "/actuator/prometheus")

So we know it's one of those conditions that's comparing a string to a non-string. Also, we know from the op value in the log that both equalities (0) and inequalities (1) are failing with this error.

opentelemetry-collector-contrib/pkg/ottl/grammar.go

Lines 121 to 132 in 22d86de

	// compareOp is the type of a comparison operator.
	type compareOp int
	// These are the allowed values of a compareOp
	const (
	EQ compareOp = iota
	NE
	LT
	LTE
	GTE
	GT
	)

[github-actions]

Pinging code owners for pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

(This is also relevant for all of the other comparisons as well, not just strings.)

[trajano]

I am thinking maybe it's the nil perhaps we it should log differently or ignore if nil == "string"

[TylerHelmuth]

This is happening because you've enabled

service:
  telemetry:
    logs:
      level: debug

At the moment it is working as intended. The log could be improved. If you want to stop the influx of logs in the meantime you should switch to

service:
  telemetry:
    logs:
      level: info

[trajano]

@TylerHelmuth thanks yes I know it's because of the debug level. I was trying to diagnose another issue which is why I was on debug mode. I just used grep -v to ignore the noise.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.