Search looks only into recent data if the configured timerange includes recent data, but does not search beyond

[its-felix]

Describe the bug
I have already reported the issue here https://community.grafana.com/t/tempo-traces-search-showing-inconsistent-results/89714 but with the findings I've made so far, this seems to be the most appropiate place for a report.

When searching for traces, the search only searches for recent data if the configured timerange includes recent data, but never searches for "cold" data even if the configured timerange would also include those.

To Reproduce

1. Prepare only cold data (i.e. no data in the ingester, but some data in the "cold" storage
2. Go to the search console
3. Configure a timerange that includes the recent data but also includes the timerange of the previously prepared "cold" data (i.e. any relative range larger than 30m)
4. Perform a search that should match the prepared data (i.e. {})
5. There will be no results (the search looks only for recent data, but there's none), even though the "cold" data should be found because the timerange+search includes them
6. Configure a timerange that excludes recent data (an absolute range that includes the previously prepared "cold" data, but not the past 30m)
7. Perform a search that should match the prepared data (i.e. {})
8. The prepared data will be found

Expected behavior
All data, given the configured timerange, search and limit, should be found, regardless of recent or non-recent ingestion.

Environment:
Grafana Cloud

Additional Context
https://community.grafana.com/t/tempo-traces-search-showing-inconsistent-results/89714

It might be related to this: https://github.com/grafana/tempo/blob/main/modules/querier/http.go#L114 because it always searches in either SearchRecent or SearchBlock but never both.

[timephy]

I have the exact same issue, i dug though the documentation for about 5h today, doing various changes... I could not find any resources related to this.

I would highly welcome help.

Observed behaviour:
Every ~1h the traces of the last ~30min disappear from the search results.

Environment:
Monolith running in Docker

[joe-elliott]

So I quickly attempted to reproduce this using https://github.com/grafana/intro-to-mlt/ but was unable to.

Steps:

• docker-compose up
• after a few minutes of generating data use docker kill to kill the agent so no new data is gathered.
• run a search over the past hour for {} and see traces returned
• wait till all blocks are removed from the ingester.
  • confirm by using docker exec -it <id> ash and doing an ls on /tmp/tempo/wal/blocks and /tmp/tempo/wal/blocks/single-tenant
  • once both of these folders are empty proceed
• run a search over the past hour for {} and see traces returned.

If you are having this issue in Grafana Cloud please raise a support case. If you can reproduce there and pass along queries performed, trace ids found/not found, we can use internal tools to debug and determine why the expected traces were not returned.

If you are seeing this locally please provide your configuration as that would help us determine what the cause would be.

[joe-elliott]

I have reproduced this on an internal cluster and will dig in tomorrow.

[joe-elliott]

Fixed in #2581

[its-felix]

Thanks. Do you know when this will be available in Grafana Cloud?

[joe-elliott]

The production rollout of this fix is ongoing now.

[deathsaber]

I am using the open source tempo-distributed chart and I have faced this issue as well with tempo v2.1.1. Do you know if this fix will be available in the next release of tempo?

[joe-elliott]

Do you know if this fix will be available in the next release of tempo?

Yes, it will be available in 2.2

[joe-elliott]

Do you know if this fix will be available in the next release of tempo?

Yes, it will be available in 2.2, but I wouldn't expect it to be in 2.1.1. I thought we only introduced it recently.