New tag name filtering doesnt filter, showing all results instead

[mentaman]

Describe the bug

Just upgraded to tempo 2.6.1 from 2.5.0 to try the new tag name filtering #3822
Trying to access tag values, with a q filter.
But it doesn't seem to filter anything.
I get all results instead of the relevant service.

Tried

non existing filter

Writing service name that doesn't exists "aaa", tried filtering, it shows many results even though I don't have any service named "aaa"

http://localhost:3200/api/v2/search/tag/.http.url/values?q=%7B%20.resource.service.name%3D%22aaa%22%20%7D
http://localhost:3200/api/v2/search/tag/.http.url/values?q=%7B%20.service.name%3D%22aaa%22%20%7D

existing filter

Writing name that does exist instead of "aaa", looking for "authorization", I saw there http.url that don't exist in the service, but belong to other services

http://localhost:3200/api/v2/search/tag/.http.url/values?q=%7B%20.service.name%3D%22authorization%22%20%7D
http://localhost:3200/api/v2/search/tag/.http.url/values?q=%7B%20.resource.service.name%3D%22authorization%22%20%7D

invalid filter

Writing invalid q doesn't throw errors
http://localhost:3200/api/v2/search/tag/.http.url/values?q=bbaaa

Seems like even if I put ?q= a completely not valid traceql, it will still show me all results instead of saying it's invalid, so seems like it ignores my q parameter completely.

I tried using it in multiple ways, encoded, not encoded, with scope or without.

To Reproduce

Steps to reproduce the behavior:

1. Start tempo 2.6.1
2. Tried filtering tag values with api
3. You'll see all results instead of filtered results

Expected behavior

Seeing only my filtered results, not all tags.

Environment:

• Infrastructure: Kubernetes
• Deployment tool: okteto

Additional Context

Seeing this log to verify it's 2.6.1

empolevel=info ts=2024-12-06T09:33:07.535701713Z caller=main.go:119 msg="Starting Tempo" version="(version=2.6.1, branch=HEAD, revision=24c5b553d)"

That's the dockerfile:

FROM grafana/tempo:2.6.1

COPY configs/tempo.yaml /etc/

CMD ["-config.file=/etc/tempo.yaml"]

[joe-elliott]

For performance reasons the filter only works on simple queries. i.e. A series of && conditions and fully scoped attributes. Unfortunately, the docs are wrong. They include an example with an unscoped attribute. cc @knylander-grafana

invalid filter

This is on purpose. We designed the feature to "fail open". In the event of an invalid query it just falls back to the old behavior.

[mentaman]

@joe-elliott thanks for the quick response! oh makes sense regarding backwards compatibility! It just makes it harder to debug when it doesn't work without an error. Maybe worth adding "strict=true" mode option to get failures?

Regarding supporting only simple filters, my filter is not considered simple? I only wanted to filter

{.service.name='myservice'}

Or the issue is that I have to add scope in my case? It's not mentioned in the documentation as you said. So I'm not sure how to fix my query. (I actually even took the exact example from docs and it didn't work for me as well)

[joe-elliott]

Maybe worth adding "strict=true" mode option to get failures?

I'd take that PR.

Or the issue is that I have to add scope in my case?

Yes. {.service.name = "myservice"} is deceptively complex. It resolves to something like { resource.service.name = "myservice" || (span.service.name = "myservice" && resource.service.name = nil)} which is not even valid TraceQL. We don't currently support = nil.

(I actually even took the exact example from docs and it didn't work for me as well)

Yes, sadly the docs are wrong. We need to update them to say that unscoped attributes are not supported for filtered tag values and change the examples.

[knylander-grafana]

I've updated the doc here: #4425

[mentaman]

Oh awesome @joe-elliott adding the scope worked thanks!!!

And thanks @knylander-grafana for the update 🙏🏻

[knylander-grafana]

You're welcome! :)

[knylander-grafana]

if you feel like your issue is resolved, please feel free to close the issue.

[mentaman]

if you feel like your issue is resolved, please feel free to close the issue.

Yes my issue is resolved thanks.
Closing the issue.