[Feature] List Partition For AMV(Part 2): Support list partition for asynchronous materialized view with non-nullable partition columns

[LiShuMing]

Why I'm doing:

What I'm doing:

• Support mv refresh for list partitioned amv;
• Support mv rewrite for list partitioned amv;

CREATE TABLE t3 (
      id BIGINT,
      province VARCHAR(64) not null,
      age SMALLINT,
      dt VARCHAR(10) not null
)
DUPLICATE KEY(id)
PARTITION BY LIST (province, dt) (
     PARTITION p1 VALUES IN (("beijing", "2024-01-01"))  ,
     PARTITION p2 VALUES IN (("guangdong", "2024-01-01")), 
     PARTITION p3 VALUES IN (("beijing", "2024-01-02"))  ,
     PARTITION p4 VALUES IN (("guangdong", "2024-01-02")) 
)
DISTRIBUTED BY RANDOM;
INSERT INTO t3 VALUES (1, 'beijing', 20, '2024-01-01'), (2, 'guangdong', 20, '2024-01-01'), (3, 'guangdong', 20, '2024-01-02');

-- test sync refresh & partition_refresh_number= 1
create materialized view test_mv1
partition by province
REFRESH DEFERRED MANUAL
distributed by hash(dt, province) buckets 10 
PROPERTIES (
"replication_num" = "1"
) 
as select dt, province, sum(age) from t3 group by dt, province;
refresh materialized view  test_mv1 with sync mode;
select * from test_mv1 order by 1, 2;
function: check_hit_materialized_view("select dt, province, sum(age) from t3 group by dt, province order by 1, 2;", "test_mv1")
select dt, province, sum(age) from t3 group by dt, province order by 1, 2;
INSERT INTO t3 VALUES (2, 'beijing', 20, '2024-01-01');
function: check_hit_materialized_view("select dt, province, sum(age) from t3 group by dt, province order by 1, 2;", "test_mv1")
select dt, province, sum(age) from t3 group by dt, province order by 1, 2;
drop materialized view test_mv1;

Fixes #46087

What type of PR is this:

• BugFix
• Feature
• Enhancement
• Refactor
• UT
• Doc
• Tool

Does this PR entail a change in behavior?

• Yes, this PR will result in a change in behavior.
• No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

• Interface/UI changes: syntax, type conversion, expression evaluation, display information
• Parameter changes: default values, similar parameters but with different default values
• Policy changes: use new policy to replace old one, functionality automatically enabled
• Feature removed
• Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

• I have added test cases for my bug fix or my new feature
• This pr needs user documentation (for new or modified features or behaviors)
  • I have added documentation for my new feature or new function
• This is a backport pr

Bugfix cherry-pick branch check:

• I have checked the version labels which the pr will be auto-backported to the target branch
  • 3.3
  • 3.2
  • 3.1
  • 3.0
  • 2.5

[sonarcloud]

Quality Gate passed

Issues
84 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[packy92]

any risk to be a time-consuming step when table or mv has a large number of partitions?

[LiShuMing]

Yep. It's possible to be time consuming.

I will add a benchmark to figure out how slow it's, and do some optimization later in the next pr.

[github-actions]

[FE Incremental Coverage Report]

✅ pass : 363 / 445 (81.57%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/scheduler/mv/MVPCTRefreshPartitioner.java	0	1	00.00%	[150]
🔵	com/starrocks/catalog/MvBaseTableUpdateInfo.java	0	2	00.00%	[73, 74]
🔵	com/starrocks/catalog/mv/MVTimelinessListPartitionArbiter.java	31	57	54.39%	[52, 53, 54, 61, 67, 68, 101, 103, 105, 106, 107, 108, 110, 111, 112, 113, 114, 115, 116, 118, 120, 121, 122, 123, 124, 125]
🔵	com/starrocks/scheduler/ExecuteOption.java	7	12	58.33%	[128, 129, 130, 132, 134]
🔵	com/starrocks/sql/common/ListPartitionDiff.java	6	10	60.00%	[39, 40, 47, 48]
🔵	com/starrocks/catalog/MvRefreshArbiter.java	2	3	66.67%	[201]
🔵	com/starrocks/scheduler/TaskRunManager.java	6	9	66.67%	[147, 149, 153]
🔵	com/starrocks/sql/common/PListAtom.java	15	21	71.43%	[39, 42, 45, 50, 55, 69]
🔵	com/starrocks/sql/analyzer/MaterializedViewAnalyzer.java	8	10	80.00%	[813, 818]
🔵	com/starrocks/sql/common/PListCellPlus.java	25	30	83.33%	[40, 79, 82, 85, 93]
🔵	com/starrocks/scheduler/mv/MVPCTRefreshListPartitioner.java	117	137	85.40%	[76, 83, 84, 138, 143, 150, 159, 171, 186, 188, 192, 208, 210, 251, 252, 267, 268, 305, 306, 307]
🔵	com/starrocks/sql/common/ListPartitionDiffer.java	113	120	94.17%	[206, 250, 251, 252, 253, 265, 266]
🔵	com/starrocks/scheduler/PartitionBasedMvRefreshProcessor.java	2	2	100.00%	[]
🔵	com/starrocks/catalog/mv/MVTimelinessArbiter.java	8	8	100.00%	[]
🔵	com/starrocks/sql/common/ListPartitionDiffResult.java	6	6	100.00%	[]
🔵	com/starrocks/sql/common/RangePartitionDiffer.java	1	1	100.00%	[]
🔵	com/starrocks/scheduler/mv/MVPCTRefreshRangePartitioner.java	3	3	100.00%	[]
🔵	com/starrocks/sql/common/PListCell.java	11	11	100.00%	[]
🔵	com/starrocks/catalog/mv/MVTimelinessRangePartitionArbiter.java	2	2	100.00%	[]

[github-actions]

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

[LiShuMing]

@Mergifyio backport branch-3.3

[mergify]

backport branch-3.3

✅ Backports have been created

• #47712 [Feature] List Partition For AMV(Part 2): Support list partition for asynchronous materialized view with non-nullable partition columns (backport #46680) has been created for branch branch-3.3 but encountered conflicts