storage: compactor produced keys are not ordered

[skyzh]

https://github.com/singularity-data/risingwave/runs/6221697790?check_suite_focus=true

[skyzh]

Maybe caused by #2122

[hzxa21]

Level1 SSTs is not ordered in the compaction task when trying to expand the L0 input SSTs set for compaction.

https://github.com/singularity-data/risingwave/blob/9dbdfb2e352210d3eba953268c9cdd98edd79664/src/meta/src/hummock/compaction/tier_compaction_picker.rs#L227

https://github.com/singularity-data/risingwave/blob/9dbdfb2e352210d3eba953268c9cdd98edd79664/src/meta/src/hummock/compaction/tier_compaction_picker.rs#L245

[BowenXiao1999]

Update: It is fixed by #2245. Not realtes to this issue.

---

I also encounter some problems, but I' m not sure it is due to Compaction (or my own code problems). When I use table id to replace the keyspace encoding of Simple Agg, I get panic when comparing key: bad_full_key_format (Intersting thing is, It do not happen for HashAgg (with group key) or HashJoin (with join key) ). The key length is only 5 (exactly the length of table id) and less than Epoch length so can not successfully split user key and epoch.

In my opinion, I would suspect that compaction may not be able to extend user key with epoch in some scenarios (e.g. Simple Agg, the keyspace is just a table_id). Previous executor_id/agg_call_idx keyspace is ok cuz it's larger than epoch length 8, now table_id only 5. I'll file a detail issue later after Wallace's PR get merged and check out with compaction guys if needed.

[skyzh]

Still not resolved. reproduced at https://github.com/singularity-data/risingwave/actions/runs/2244511049

[hzxa21]

Digging into the codes and the logs a little bit. The bug is caused by missing to select all overlapping L1 files as the input sst for compaction, which then causes overlaps in L1 SSTs.

Assume

SSTs:
L0: [2, 3]
L1: [5, 6]

Key Range:
|____2____|             |____3____| 
      |__5__| |__6__|

Here is what happned:

1. Initially we put L0: [2] in the compaction input SST set
2. We expand input to L0: [2], L1: [5] since 5 overlaps with 2
3. We expand input to L0: [2, 3], L1: [5] since the input set size is too small
4. We won't expand input any more since we only check overlap with the newly added SST:
   https://github.com/singularity-data/risingwave/blob/9dbdfb2e352210d3eba953268c9cdd98edd79664/src/meta/src/hummock/compaction/tier_compaction_picker.rs#L246

The expected input set is L0: [2, 3], L1: [5, 6]. The root cause is we didn't check overlap on the full key range on the input set.

[hzxa21]

Debugging tips:

• Check compactor logs to find which task cause the panic (grep "Ready to handle compaction task")
• Check meta logs to find the compaction info (grep "Compaction task id")
• Check meta logs to find the KeyRange of an SST (grep "KeyRange")

[hzxa21]

Fixed in #2242