[Enhancement] improve statistics strategy for insert overwrite #50417

murphyatwork · 2024-08-29T08:54:39Z

Why I'm doing:

INSERT OVERWRITE would create a new partition with different id, and with version=2, which breaks the hypothesis of current load-triggered statistics collection.

What I'm doing:

INSERT OVERWRITE should not re-collect the statistics but copy the existing statistics instead.
if changed ratio of a table/partition is less than 10%, we don't trigger the collection

Fixes #issue

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

fe/fe-core/src/main/java/com/starrocks/sql/optimizer/statistics/InMemoryStatisticStorage.java

fe/fe-core/src/main/java/com/starrocks/common/Config.java

murphyatwork · 2024-09-24T23:27:27Z

@Mergifyio rebase

mergify · 2024-09-24T23:27:35Z

rebase

❌ Unable to rebase: user `murphyatwork` is unknown.

Please make sure murphyatwork has logged in Mergify dashboard.

Signed-off-by: Murphy <[email protected]>

sonarcloud · 2024-10-08T09:05:53Z

Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

github-actions · 2024-10-08T11:10:50Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-10-08T11:11:27Z

[FE Incremental Coverage Report]

✅ pass : 226 / 244 (92.62%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/sql/optimizer/statistics/StatisticStorage.java	1	2	50.00%	[48]
🔵	com/starrocks/statistic/StatisticsCollectionTrigger.java	134	149	89.93%	[132, 157, 158, 159, 197, 198, 199, 211, 212, 220, 227, 228, 229, 230, 293]
🔵	com/starrocks/listener/LoadJobStatsListener.java	12	13	92.31%	[74]
🔵	com/starrocks/qe/DmlType.java	14	15	93.33%	[45]
🔵	com/starrocks/catalog/OlapTable.java	1	1	100.00%	[]
🔵	com/starrocks/common/Config.java	1	1	100.00%	[]
🔵	com/starrocks/transaction/InsertOverwriteJobStats.java	20	20	100.00%	[]
🔵	com/starrocks/statistic/StatisticExecutor.java	7	7	100.00%	[]
🔵	com/starrocks/statistic/StatisticUtils.java	3	3	100.00%	[]
🔵	com/starrocks/qe/StmtExecutor.java	4	4	100.00%	[]
🔵	com/starrocks/load/InsertOverwriteJobRunner.java	14	14	100.00%	[]
🔵	com/starrocks/statistic/FullStatisticsCollectJob.java	12	12	100.00%	[]
🔵	com/starrocks/listener/GlobalLoadJobListenerBus.java	2	2	100.00%	[]
🔵	com/starrocks/transaction/TableCommitInfo.java	1	1	100.00%	[]

github-actions · 2024-10-08T11:11:36Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-10-09T06:19:36Z

@Mergifyio backport branch-3.3

mergify · 2024-10-09T06:19:53Z

backport branch-3.3

✅ Backports have been created

#51656 [Enhancement] improve statistics strategy for insert overwrite (backport #50417) has been created for branch branch-3.3 but encountered conflicts

Signed-off-by: Murphy <[email protected]> (cherry picked from commit 5126313) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java # fe/fe-core/src/main/java/com/starrocks/statistic/StatisticUtils.java

Signed-off-by: Murphy <[email protected]> (cherry picked from commit 5126313) Signed-off-by: Murphy <[email protected]> # Conflicts: # fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java # fe/fe-core/src/main/java/com/starrocks/statistic/StatisticUtils.java

wangsimo0 · 2024-10-09T13:20:57Z

ignore backport check: 3.3.5

Signed-off-by: Murphy <[email protected]> (cherry picked from commit 5126313) Signed-off-by: Murphy <[email protected]>

…ort #50417) (#51656) Signed-off-by: Murphy <[email protected]> Co-authored-by: Murphy <[email protected]>

…ocks#50417) Signed-off-by: Murphy <[email protected]>

…ocks#50417) Signed-off-by: Murphy <[email protected]> Signed-off-by: zhiminr.ren <[email protected]>

murphyatwork requested review from a team as code owners August 29, 2024 08:54

mergify bot assigned murphyatwork Aug 29, 2024

github-actions bot added the 3.3 label Aug 29, 2024

murphyatwork force-pushed the murphy_fix_overwrite_statistics branch from a47fced to 8cff413 Compare September 3, 2024 03:13

Seaven reviewed Sep 3, 2024

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/sql/optimizer/statistics/InMemoryStatisticStorage.java Outdated Show resolved Hide resolved

Seaven reviewed Sep 3, 2024

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/common/Config.java Outdated Show resolved Hide resolved

packy92 previously approved these changes Sep 5, 2024

View reviewed changes

murphyatwork dismissed packy92’s stale review via 66c35e3 September 6, 2024 07:16

murphyatwork force-pushed the murphy_fix_overwrite_statistics branch from f59c9c0 to 66c35e3 Compare September 6, 2024 07:16

Seaven previously approved these changes Sep 6, 2024

View reviewed changes

murphyatwork dismissed Seaven’s stale review via ea4687d September 9, 2024 07:48

murphyatwork force-pushed the murphy_fix_overwrite_statistics branch 3 times, most recently from 9b04c41 to 8524579 Compare September 16, 2024 01:56

murphyatwork force-pushed the murphy_fix_overwrite_statistics branch 2 times, most recently from 970d3da to 4407df9 Compare September 23, 2024 06:49

murphyatwork force-pushed the murphy_fix_overwrite_statistics branch 3 times, most recently from c120005 to fe8c8d8 Compare September 26, 2024 06:42

murphyatwork added 7 commits October 8, 2024 10:58

optimize wait strategy for statistics collection for first load

0132fef

Signed-off-by: Murphy <[email protected]>

fix race condition

a3e566f

Signed-off-by: Murphy <[email protected]>

improve statistics collection for overwrite

792a489

Signed-off-by: Murphy <[email protected]>

refresh cache after updating partition statistic

f55851f

Signed-off-by: Murphy <[email protected]>

add comment

4fd19ca

Signed-off-by: Murphy <[email protected]>

add test

5f5b604

Signed-off-by: Murphy <[email protected]>

fix

7388254

Signed-off-by: Murphy <[email protected]>

murphyatwork added 3 commits October 8, 2024 10:59

fix conflict

80c9942

Signed-off-by: Murphy <[email protected]>

fix test

48b11a9

Signed-off-by: Murphy <[email protected]>

skip trigger on follower && replay

f0d66c6

Signed-off-by: Murphy <[email protected]>

murphyatwork force-pushed the murphy_fix_overwrite_statistics branch from 0e965da to f0d66c6 Compare October 8, 2024 02:59

Seaven approved these changes Oct 8, 2024

View reviewed changes

HangyuanLiu approved these changes Oct 8, 2024

View reviewed changes

meegoo approved these changes Oct 9, 2024

View reviewed changes

murphyatwork merged commit 5126313 into StarRocks:main Oct 9, 2024
51 of 53 checks passed

github-actions bot removed the 3.3 label Oct 9, 2024

mergify bot mentioned this pull request Oct 9, 2024

[Enhancement] improve statistics strategy for insert overwrite (backport #50417) #51656

Merged

42 tasks

murphyatwork added a commit that referenced this pull request Oct 10, 2024

[Enhancement] improve statistics strategy for insert overwrite (#50417)

66d1aa1

Signed-off-by: Murphy <[email protected]> (cherry picked from commit 5126313) Signed-off-by: Murphy <[email protected]>

wanpengfei-git pushed a commit that referenced this pull request Oct 10, 2024

[Enhancement] improve statistics strategy for insert overwrite (backp…

167e2f1

…ort #50417) (#51656) Signed-off-by: Murphy <[email protected]> Co-authored-by: Murphy <[email protected]>

github-actions bot added the 3.3-merged label Oct 10, 2024

ZiheLiu pushed a commit to ZiheLiu/starrocks that referenced this pull request Oct 31, 2024

[Enhancement] improve statistics strategy for insert overwrite (StarR…

5f377b6

…ocks#50417) Signed-off-by: Murphy <[email protected]>

renzhimin7 pushed a commit to renzhimin7/starrocks that referenced this pull request Nov 7, 2024

[Enhancement] improve statistics strategy for insert overwrite (StarR…

4779d2b

…ocks#50417) Signed-off-by: Murphy <[email protected]> Signed-off-by: zhiminr.ren <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] improve statistics strategy for insert overwrite #50417

[Enhancement] improve statistics strategy for insert overwrite #50417

murphyatwork commented Aug 29, 2024 •

edited

Loading

murphyatwork commented Sep 24, 2024

mergify bot commented Sep 24, 2024

sonarcloud bot commented Oct 8, 2024

github-actions bot commented Oct 8, 2024

github-actions bot commented Oct 8, 2024

github-actions bot commented Oct 8, 2024

github-actions bot commented Oct 9, 2024

mergify bot commented Oct 9, 2024 •

edited

Loading

wangsimo0 commented Oct 9, 2024

[Enhancement] improve statistics strategy for insert overwrite #50417

[Enhancement] improve statistics strategy for insert overwrite #50417

Conversation

murphyatwork commented Aug 29, 2024 • edited Loading

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

murphyatwork commented Sep 24, 2024

mergify bot commented Sep 24, 2024

❌ Unable to rebase: user murphyatwork is unknown.

sonarcloud bot commented Oct 8, 2024

Quality Gate failed

github-actions bot commented Oct 8, 2024

[Java-Extensions Incremental Coverage Report]

github-actions bot commented Oct 8, 2024

[FE Incremental Coverage Report]

file detail

github-actions bot commented Oct 8, 2024

[BE Incremental Coverage Report]

github-actions bot commented Oct 9, 2024

mergify bot commented Oct 9, 2024 • edited Loading

✅ Backports have been created

wangsimo0 commented Oct 9, 2024

murphyatwork commented Aug 29, 2024 •

edited

Loading

❌ Unable to rebase: user `murphyatwork` is unknown.

mergify bot commented Oct 9, 2024 •

edited

Loading