[BugFix] fix partition min/max prune with null-value partitions #53235

murphyatwork · 2024-11-27T06:10:43Z

Why I'm doing:

create table t2(user_id int, dt datetime) partition by date_trunc('day', dt);
insert into t2 values (1, '2024-10-05 01:01:01'), (2, '2024-10-06 02:02:02'), (3, '2024-10-07 03:03:03'), (4, NULL);
select min(dt), max(dt) from t2;

For this case, we need to rule out NULL partition when doing min/max partition pruning, because min/max aggregation function itself also rule out NULL values.

What I'm doing:

Fixes #issue

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

fe/fe-core/src/main/java/com/starrocks/catalog/RangePartitionInfo.java

fe/fe-core/src/main/java/com/starrocks/catalog/ListPartitionInfo.java

...n/java/com/starrocks/sql/optimizer/rule/transformation/PartitionColumnMinMaxRewriteRule.java

wyb · 2024-12-02T02:28:40Z

fe/fe-core/src/main/java/com/starrocks/catalog/ListPartitionInfo.java

+     * @return
+     */
+    @Override
+    public List<Long> getNullValuePartitions() {


why not use Set<Long>?

Signed-off-by: Murphy <[email protected]>

sonarcloud · 2024-12-03T02:06:11Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions · 2024-12-03T11:26:06Z

[FE Incremental Coverage Report]

✅ pass : 24 / 27 (88.89%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/catalog/PartitionInfo.java	0	1	00.00%	[263]
🔵	com/starrocks/sql/optimizer/rule/transformation/PartitionColumnMinMaxRewriteRule.java	10	11	90.91%	[253]
🔵	com/starrocks/catalog/ListPartitionInfo.java	10	11	90.91%	[634]
🔵	com/starrocks/catalog/RangePartitionInfo.java	4	4	100.00%	[]

github-actions · 2024-12-03T11:26:17Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-12-03T11:26:29Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-12-03T12:45:43Z

@Mergifyio backport branch-3.4

github-actions · 2024-12-03T12:45:44Z

@Mergifyio backport branch-3.3

github-actions · 2024-12-03T12:45:45Z

@Mergifyio backport branch-3.2

mergify · 2024-12-03T12:46:06Z

backport branch-3.4

✅ Backports have been created

#53495 [BugFix] fix partition min/max prune with null-value partitions (backport #53235) has been created for branch branch-3.4

mergify · 2024-12-03T12:46:07Z

backport branch-3.3

✅ Backports have been created

#53496 [BugFix] fix partition min/max prune with null-value partitions (backport #53235) has been created for branch branch-3.3

mergify · 2024-12-03T12:46:09Z

backport branch-3.2

✅ Backports have been created

#53497 [BugFix] fix partition min/max prune with null-value partitions (backport #53235) has been created for branch branch-3.2 but encountered conflicts

Signed-off-by: Murphy <[email protected]> (cherry picked from commit 5bad4af)

Signed-off-by: Murphy <[email protected]> (cherry picked from commit 5bad4af) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/catalog/ListPartitionInfo.java # fe/fe-core/src/main/java/com/starrocks/catalog/RangePartitionInfo.java # fe/fe-core/src/main/java/com/starrocks/sql/optimizer/rule/transformation/PartitionColumnMinMaxRewriteRule.java # fe/fe-core/src/test/java/com/starrocks/sql/plan/PartitionPruneTest.java

…port #53235) (#53496) Co-authored-by: Murphy <[email protected]>

…port #53235) (#53495) Co-authored-by: Murphy <[email protected]>

murphyatwork requested review from a team as code owners November 27, 2024 06:10

mergify bot assigned murphyatwork Nov 27, 2024

starrocks-cr bot reviewed Nov 27, 2024

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/catalog/RangePartitionInfo.java Show resolved Hide resolved

starrocks-cr bot reviewed Nov 27, 2024

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/catalog/ListPartitionInfo.java Show resolved Hide resolved

starrocks-cr bot reviewed Nov 27, 2024

View reviewed changes

...n/java/com/starrocks/sql/optimizer/rule/transformation/PartitionColumnMinMaxRewriteRule.java Show resolved Hide resolved

Seaven previously approved these changes Nov 27, 2024

View reviewed changes

LiShuMing previously approved these changes Nov 27, 2024

View reviewed changes

murphyatwork dismissed stale reviews from LiShuMing and Seaven via c596f66 November 27, 2024 07:32

murphyatwork force-pushed the murphy_fix_partition_prune_null branch 2 times, most recently from d11e0d3 to cf51d7e Compare November 28, 2024 07:58

github-actions bot added 3.4 3.3 3.2 labels Nov 28, 2024

LiShuMing previously approved these changes Nov 29, 2024

View reviewed changes

murphyatwork enabled auto-merge (squash) November 29, 2024 07:28

wyb reviewed Dec 2, 2024

View reviewed changes

murphyatwork dismissed LiShuMing’s stale review via 48e5e3f December 2, 2024 02:41

murphyatwork force-pushed the murphy_fix_partition_prune_null branch from f410d68 to 48e5e3f Compare December 2, 2024 02:41

wyb previously approved these changes Dec 2, 2024

View reviewed changes

murphyatwork dismissed wyb’s stale review via 14328c8 December 2, 2024 03:02

murphyatwork force-pushed the murphy_fix_partition_prune_null branch from 48e5e3f to 14328c8 Compare December 2, 2024 03:02

murphyatwork added 6 commits December 3, 2024 10:00

fix partition min/max prune with null-value partitions

549a133

Signed-off-by: Murphy <[email protected]>

sql test

6f50381

Signed-off-by: Murphy <[email protected]>

give up null partitions

c891f89

Signed-off-by: Murphy <[email protected]>

fix test

406cc24

Signed-off-by: Murphy <[email protected]>

add test

76a8399

Signed-off-by: Murphy <[email protected]>

refactor to set

d9982da

Signed-off-by: Murphy <[email protected]>

murphyatwork force-pushed the murphy_fix_partition_prune_null branch from 14328c8 to d9982da Compare December 3, 2024 02:00

Seaven approved these changes Dec 3, 2024

View reviewed changes

LiShuMing approved these changes Dec 3, 2024

View reviewed changes

wyb approved these changes Dec 3, 2024

View reviewed changes

murphyatwork merged commit 5bad4af into StarRocks:main Dec 3, 2024
51 checks passed

github-actions bot removed the 3.4 label Dec 3, 2024

github-actions bot removed the 3.3 label Dec 3, 2024

github-actions bot removed the 3.2 label Dec 3, 2024

mergify bot pushed a commit that referenced this pull request Dec 3, 2024

[BugFix] fix partition min/max prune with null-value partitions (#53235)

dd2a232

Signed-off-by: Murphy <[email protected]> (cherry picked from commit 5bad4af)

mergify bot mentioned this pull request Dec 3, 2024

[BugFix] fix partition min/max prune with null-value partitions (backport #53235) #53495

Merged

42 tasks

mergify bot pushed a commit that referenced this pull request Dec 3, 2024

[BugFix] fix partition min/max prune with null-value partitions (#53235)

0842aee

Signed-off-by: Murphy <[email protected]> (cherry picked from commit 5bad4af)

This was referenced Dec 3, 2024

[BugFix] fix partition min/max prune with null-value partitions (backport #53235) #53496

Merged

[BugFix] fix partition min/max prune with null-value partitions (backport #53235) #53497

Closed

wanpengfei-git pushed a commit that referenced this pull request Dec 4, 2024

[BugFix] fix partition min/max prune with null-value partitions (back…

ef2175f

…port #53235) (#53496) Co-authored-by: Murphy <[email protected]>

github-actions bot added 3.3-merged 3.4 3.3 labels Dec 4, 2024

wanpengfei-git pushed a commit that referenced this pull request Dec 4, 2024

[BugFix] fix partition min/max prune with null-value partitions (back…

e681576

…port #53235) (#53495) Co-authored-by: Murphy <[email protected]>

github-actions bot added the 3.4-merged label Dec 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BugFix] fix partition min/max prune with null-value partitions #53235

[BugFix] fix partition min/max prune with null-value partitions #53235

murphyatwork commented Nov 27, 2024 •

edited

Loading

wyb Dec 2, 2024

murphyatwork Dec 2, 2024

sonarcloud bot commented Dec 3, 2024

github-actions bot commented Dec 3, 2024

github-actions bot commented Dec 3, 2024

github-actions bot commented Dec 3, 2024

github-actions bot commented Dec 3, 2024

github-actions bot commented Dec 3, 2024

github-actions bot commented Dec 3, 2024

mergify bot commented Dec 3, 2024 •

edited

Loading

mergify bot commented Dec 3, 2024 •

edited

Loading

mergify bot commented Dec 3, 2024 •

edited

Loading

[BugFix] fix partition min/max prune with null-value partitions #53235

[BugFix] fix partition min/max prune with null-value partitions #53235

Conversation

murphyatwork commented Nov 27, 2024 • edited Loading

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

wyb Dec 2, 2024

Choose a reason for hiding this comment

murphyatwork Dec 2, 2024

Choose a reason for hiding this comment

sonarcloud bot commented Dec 3, 2024

Quality Gate passed

github-actions bot commented Dec 3, 2024

[FE Incremental Coverage Report]

file detail

github-actions bot commented Dec 3, 2024

[Java-Extensions Incremental Coverage Report]

github-actions bot commented Dec 3, 2024

[BE Incremental Coverage Report]

github-actions bot commented Dec 3, 2024

github-actions bot commented Dec 3, 2024

github-actions bot commented Dec 3, 2024

mergify bot commented Dec 3, 2024 • edited Loading

✅ Backports have been created

mergify bot commented Dec 3, 2024 • edited Loading

✅ Backports have been created

mergify bot commented Dec 3, 2024 • edited Loading

✅ Backports have been created

murphyatwork commented Nov 27, 2024 •

edited

Loading

mergify bot commented Dec 3, 2024 •

edited

Loading

mergify bot commented Dec 3, 2024 •

edited

Loading

mergify bot commented Dec 3, 2024 •

edited

Loading