queue_depth default not working

[mrkwtz]

Describe the bug
It seems like the storage.trace.pool.queue_depth default config in https://github.com/grafana/tempo/blob/v1.3.2/tempodb/pool/config.go#L12 which should be 10k is not applied. When we don't set it, it seems to default to 200 and causes a lot of error querying store in Querier.FindTraceByID: queue doesn't have room for X jobs errors.

That is our tempodb_work_queue_max metric value when we don't override the defaults

That is our tempodb_work_queue_max metric value when we set storage.trace.pool.queue_depth explicitly to 10k

As soon as we set the configuration explicitly the error querying store in Querier.FindTraceByID: queue doesn't have room for X jobs errors disappear.

Expected behavior
storage.trace.pool.queue_depth should default to 10k as documented

Environment:

• Tempo version: 1.3.2
• Infrastructure: Kubernetes
• Deployment tool: helm

[joe-elliott]

Thanks for the issue! The actual default config is set here:
https://github.com/grafana/tempo/blob/main/modules/storage/config.go#L83

Although our docs say the defaults are the values from the code that you linked:
https://github.com/grafana/tempo/blob/main/docs/tempo/website/configuration/_index.md?plain=1#L643

So either we need to fix the defaults or fix the docs. Do you have strong feelings either way on what the defaults should be?

[mrkwtz]

Ahh good to know :) I would vote for fixing the defaults. Like I said with the 200 default we continuously run into the above mentioned errors and I would say our setup is not that big. If the size of the queue_depth does not cause any overhead I guess 10k should be fine

[joe-elliott]

Yeah, I agree that that default has not aged well. I appreciate you bringing this to our attention. The fix is in!