Compaction sometimes fails with "failed to find a lowest object in compaction"

[joe-elliott]

Describe the bug
After installing Tempo 1.2 some users are seeing the following error message during compaction which results in a failed compaction attempt.

msg=“error during compaction cycle” err=“error iterating input blocks: failed to find a lowest object in compaction”

This is due to 0 length traces showing up in blocks.

To Reproduce
Steps to reproduce the behavior:

1. Start Tempo 1.2
2. Use it
3. Maybe compactions start failing.

Expected behavior
Compaction to work.

[BitProcessor]

Seeing the same since 1.2.0:

Message like these, over and over:

level=error ts=2021-11-11T20:42:22.223983257Z caller=compactor.go:110 msg="error during compaction cycle" err="error iterating input blocks: failed to find a lowest object in compaction"
level=info ts=2021-11-11T20:42:22.227494482Z caller=compactor.go:104 msg="Compacting hash" hashString=single-tenant-454583
level=info ts=2021-11-11T20:42:22.229293825Z caller=compactor.go:150 msg="compacting block" block="&{Version:v2 BlockID:96570569-cbc3-478c-bbfb-0e30b834eb72 MinID:[0 2 70 95 66 139 228 161 241 159 6 118 191 138 224 207] MaxID:[255 255 164 61 224 31 74 85 42 215 58 193 250 30 142 249] TenantID:single-tenant StartTime:2021-11-0
9 22:17:35.750929025 +0000 UTC EndTime:2021-11-09 23:17:45.750513096 +0000 UTC TotalObjects:38751 Size:10171166 CompactionLevel:0 Encoding:zstd IndexPageSize:256000 TotalRecords:212 DataEncoding:v1 BloomShardCount:1}"
level=info ts=2021-11-11T20:42:22.309175246Z caller=compactor.go:150 msg="compacting block" block="&{Version:v2 BlockID:56307324-faab-423f-ba90-d66b46bbafd0 MinID:[0 2 61 214 45 116 207 214 205 208 96 78 117 240 202 190] MaxID:[255 252 123 149 143 49 139 217 110 255 52 254 135 104 118 2] TenantID:single-tenant StartTime:2021-
11-09 22:14:12.351512661 +0000 UTC EndTime:2021-11-09 23:14:12.355201773 +0000 UTC TotalObjects:41455 Size:10958996 CompactionLevel:0 Encoding:zstd IndexPageSize:256000 TotalRecords:227 DataEncoding:v1 BloomShardCount:1}"
level=info ts=2021-11-11T20:42:22.331181992Z caller=compactor.go:150 msg="compacting block" block="&{Version:v2 BlockID:79ba8c58-a399-4e01-a4bb-286f1bf8e5c2 MinID:[0 1 228 205 42 72 141 67 189 145 160 99 168 112 37 194] MaxID:[255 255 171 73 6 60 74 222 214 110 231 184 50 24 190 225] TenantID:single-tenant StartTime:2021-11-0
9 22:12:21.053164108 +0000 UTC EndTime:2021-11-09 23:12:31.052552491 +0000 UTC TotalObjects:43929 Size:11576949 CompactionLevel:0 Encoding:zstd IndexPageSize:256000 TotalRecords:242 DataEncoding:v1 BloomShardCount:1}"

[joe-elliott]

Thanks for the report. 👍
We are actively working on a 1.2.1 to fix the issue.