TraceQL: unequal span attribute filter filters traces that do not contain the span attribute

[matthias-huber]

Describe the bug
A typical usecase of TraceQL is to find all traces for an app that do not match specific span attributes, e.g. all traces that do not contain the http path /my/uri. Such a TraceQL could be

{resource.app = "my-app" && span.http.target != "/my/uri"}

This works if all traces of the app 'my-app' contain a span.attribute 'http.target'. Unfortunately, traces that do not contain that span.attribute are also filtered out and aren't searchable anymore without a workaround like

{resource.app = "my-app" && (span.http.target !="/my/uri" || span.existing.attribute != "")}

To Reproduce
Steps to reproduce the behavior:

1. Start Tempo (SHA or version)
2. Insert at least two traces, one with the span attribute and one without a span attribute.
3. Use the above search a try to find the span without the attribute

Expected behavior
From my perspective, filtering should not filter traces that do not have the span.attribute.
Alternatively, TraceQL should provide a way to exclude traces from the result that match a condition. I am not aware of this functionality.

Environment:

• Infrastructure: [e.g., Kubernetes, bare-metal, laptop]
• Deployment tool: [e.g., helm, jsonnet]

Additional Context

[joe-elliott]

Thanks for the report. These may be relevant:

Notting spansets
Attribute existence

From my perspective, filtering should not filter traces that do not have the span.attribute.

The general thought is that span.attribute != "foo" should include those spans that have span.attribute but don't equal foo as well as all spans that don't have that attribute.

We are currently doing a second round of TraceQL designs and this issue (along with the two linked ones) all seem to be pointing toward similar functionality. Thanks for filing the issue. We will take it into consideration while moving TraceQL forward.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days.
The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity.
Please apply keepalive label to exempt this Issue.