[Enhancement] active mv automatically #32829

murphyatwork · 2023-10-16T08:28:50Z

Fixes #issue

Introduce a daemon thread MVActiveChecker:

Try to activate MV, if it's not set to inactive manually
The default check interval is 1 minute, user could modify it through admin set frontend config('mv_active_checker_interval_seconds'='300')

Use case:

The base table of MV could be dropped and created everyday, for Hive external table
Without this mechanism, user need to activate it in the workflow, which is tedious

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

starrocks-cr · 2023-10-16T08:29:46Z

fe/fe-core/src/main/java/com/starrocks/scheduler/MVActiveChecker.java

+            ConnectContext.remove();
+        }
+    }
+}


Here are some suggested improvements:

Exception handling:
In several places, the code simply logs the exception with a warning and continues to run. This can make debugging harder since the cause of the problem may get lost. You should consider whether it's appropriate to deal better with exceptions.

Excess object creation:
Inside the method tryToActivate(MaterializedView mv), you create a new TableName object only to convert it to a string. If the toString() function isn't costly in terms of resources, this might not be an issue, but if it is, consider using another method to construct the full table name from dbName and mv.getName().

Running through databases and tables:
You might want to consider more effective data structures or methods for searching through your databases and tables. Currently, the complexity is proportional to the number of databases multiplied by the number of tables, which will lead to performance issues as these numbers increase.

Inconsistent usage of Optional:
In tryToActivate(MaterializedView mv), you’re accessing dbName via Optional. But in other parts of the code, null check isn't done explicitly nor with Optional. Having consistent use of Optional is good practice to avoid NullPointerException.

Thread Safety:
It's unclear whether the code is running in a multithreaded environment or not. If it does, you need to ensure thread safety especially when manipulating shared resources. Consider potential concurrency issues for methods like process(), runAfterCatalogReady(), and tryToActivate().

Code comments:
Although there are relevant comments and it's always beneficial to comment your code, remember that the code itself should be mostly self-explanatory. Aim to write descriptive variable, method, and class names so that comments are needed less for explaining what happens and more for why certain decisions were made.

Please ensure to consider these suggestions taking into account the entire application context since some may not be applicable depending on intended behavior of the software, environment among other factors.

murphyatwork · 2023-10-16T11:02:56Z

@Mergifyio rebase

mergify · 2023-10-16T11:03:17Z

rebase

✅ Branch has been successfully rebased

ABingHuang · 2023-10-17T03:24:34Z

fe/fe-core/src/main/java/com/starrocks/scheduler/MVActiveChecker.java

+    private void process() {
+        Collection<Database> dbs = GlobalStateMgr.getCurrentState().getIdToDb().values();
+        for (Database db : CollectionUtils.emptyIfNull(dbs)) {
+            for (Table table : CollectionUtils.emptyIfNull(db.getTables())) {


what will happen if dropping the db during processing.

the table list is got from a ConcurrentHashMap

tryToActivate will verify the existence of database again, throw exception if database doesn't exist

ABingHuang · 2023-10-17T03:25:11Z

can we add a logic to active mv during mv refresh?

Signed-off-by: Murphy <[email protected]>

sonarcloud · 2023-10-17T06:48:24Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
9 Code Smells

0.0% Coverage
0.0% Duplication

wanpengfei-git · 2023-10-17T07:46:23Z

[BE Incremental Coverage Report]

😍 pass : 0 / 0 (0%)

wanpengfei-git · 2023-10-17T07:46:29Z

[FE Incremental Coverage Report]

😍 pass : 44 / 51 (86.27%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/scheduler/MVActiveChecker.java	36	43	83.72%	[53, 54, 87, 88, 104, 105, 106]
🔵	com/starrocks/scheduler/PartitionBasedMvRefreshProcessor.java	3	3	100.00%	[]
🔵	com/starrocks/common/Config.java	1	1	100.00%	[]
🔵	com/starrocks/alter/AlterJobMgr.java	1	1	100.00%	[]
🔵	com/starrocks/server/GlobalStateMgr.java	3	3	100.00%	[]

wanpengfei-git · 2023-10-17T12:10:08Z

@Mergifyio backport branch-3.2

wanpengfei-git · 2023-10-17T12:10:11Z

@Mergifyio backport branch-3.1

mergify · 2023-10-17T12:10:13Z

backport branch-3.2

✅ Backports have been created

#32981 [Enhancement] active mv automatically (backport #32829) has been created for branch branch-3.2

wanpengfei-git · 2023-10-17T12:10:13Z

@Mergifyio backport branch-3.0

mergify · 2023-10-17T12:10:15Z

backport branch-3.1

✅ Backports have been created

#32982 [Enhancement] active mv automatically (backport #32829) has been created for branch branch-3.1 but encountered conflicts

mergify · 2023-10-17T12:10:20Z

backport branch-3.0

✅ Backports have been created

#32983 [Enhancement] active mv automatically (backport #32829) has been created for branch branch-3.0 but encountered conflicts

Signed-off-by: Murphy <[email protected]> (cherry picked from commit c1e3a1e)

Signed-off-by: Murphy <[email protected]> (cherry picked from commit c1e3a1e) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/alter/AlterJobMgr.java # fe/fe-core/src/main/java/com/starrocks/server/GlobalStateMgr.java # fe/fe-core/src/test/java/com/starrocks/analysis/AlterMaterializedViewTest.java

Signed-off-by: Murphy <[email protected]> (cherry picked from commit c1e3a1e) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/alter/AlterJobMgr.java # fe/fe-core/src/main/java/com/starrocks/common/Config.java # fe/fe-core/src/main/java/com/starrocks/server/GlobalStateMgr.java # fe/fe-core/src/test/java/com/starrocks/analysis/AlterMaterializedViewTest.java # fe/fe-core/src/test/java/com/starrocks/scheduler/PartitionBasedMvRefreshProcessorTest.java

Signed-off-by: Murphy <[email protected]> (cherry picked from commit c1e3a1e)

Signed-off-by: Murphy <[email protected]> (cherry picked from commit c1e3a1e) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/alter/AlterJobMgr.java # fe/fe-core/src/main/java/com/starrocks/server/GlobalStateMgr.java # fe/fe-core/src/test/java/com/starrocks/analysis/AlterMaterializedViewTest.java

murphyatwork · 2024-01-05T05:54:34Z

@Mergifyio backport branch-2.5

mergify · 2024-01-05T05:54:38Z

backport branch-2.5

✅ Backports have been created

#38521 [Enhancement] active mv automatically (backport #32829) has been created for branch branch-2.5 but encountered conflicts

Signed-off-by: Murphy <[email protected]> (cherry picked from commit c1e3a1e) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/alter/Alter.java # fe/fe-core/src/main/java/com/starrocks/common/Config.java # fe/fe-core/src/main/java/com/starrocks/server/GlobalStateMgr.java # fe/fe-core/src/test/java/com/starrocks/analysis/AlterMaterializedViewTest.java # fe/fe-core/src/test/java/com/starrocks/scheduler/PartitionBasedMvRefreshProcessorTest.java

Signed-off-by: Murphy <[email protected]> (cherry picked from commit c1e3a1e)

Signed-off-by: Murphy <[email protected]> (cherry picked from commit c1e3a1e) Signed-off-by: Murphy <[email protected]>

Signed-off-by: Murphy <[email protected]> Co-authored-by: Murphy <[email protected]> Co-authored-by: Murphy <[email protected]>

mergify bot assigned murphyatwork Oct 16, 2023

github-actions bot added 3.2 3.1 3.0 labels Oct 16, 2023

starrocks-cr bot reviewed Oct 16, 2023

View reviewed changes

murphyatwork added the sig/materialized view label Oct 16, 2023

murphyatwork changed the title ~~[Enhancement] introduce a daemon thread MVActiveChecker to fix inactive MV~~ [Enhancement] active mv automatically Oct 16, 2023

murphyatwork force-pushed the murphy_opt_mv_active_checker branch from 3b9b365 to b13480c Compare October 16, 2023 11:03

satanson previously approved these changes Oct 17, 2023

View reviewed changes

ABingHuang reviewed Oct 17, 2023

View reviewed changes

murphyatwork added 3 commits October 17, 2023 13:19

introduce a daemon thread MVActiveChecker to fix inactive MV

4529c5d

Signed-off-by: Murphy <[email protected]>

manually inactive

cb057bb

Signed-off-by: Murphy <[email protected]>

activate before refreshing

98aeef2

Signed-off-by: Murphy <[email protected]>

murphyatwork dismissed satanson’s stale review via 98aeef2 October 17, 2023 05:58

murphyatwork force-pushed the murphy_opt_mv_active_checker branch from b13480c to 98aeef2 Compare October 17, 2023 05:58

fix test

0333031

Signed-off-by: Murphy <[email protected]>

LiShuMing approved these changes Oct 17, 2023

View reviewed changes

satanson approved these changes Oct 17, 2023

View reviewed changes

satanson merged commit c1e3a1e into StarRocks:main Oct 17, 2023
47 checks passed

github-actions bot removed the 3.2 label Oct 17, 2023

github-actions bot removed the 3.1 label Oct 17, 2023

github-actions bot removed the 3.0 label Oct 17, 2023

mergify bot pushed a commit that referenced this pull request Oct 17, 2023

[Enhancement] active mv automatically (#32829)

0c4fcb1

Signed-off-by: Murphy <[email protected]> (cherry picked from commit c1e3a1e)

mergify bot mentioned this pull request Oct 17, 2023

[Enhancement] active mv automatically (backport #32829) #32981

Merged

This was referenced Oct 17, 2023

[Enhancement] active mv automatically (backport #32829) #32982

Merged

[Enhancement] active mv automatically (backport #32829) #32983

Merged

wanpengfei-git pushed a commit that referenced this pull request Oct 17, 2023

[Enhancement] active mv automatically (#32829)

6c56ec2

Signed-off-by: Murphy <[email protected]> (cherry picked from commit c1e3a1e)

github-actions bot added 3.2 3.1 3.0 2.5 labels Jan 5, 2024

mergify bot mentioned this pull request Jan 5, 2024

[Enhancement] active mv automatically (backport #32829) #38521

Merged

18 tasks

murphyatwork added a commit that referenced this pull request Jan 5, 2024

[Enhancement] active mv automatically (#32829)

4ca15ee

Signed-off-by: Murphy <[email protected]> (cherry picked from commit c1e3a1e)

murphyatwork added a commit that referenced this pull request Jan 5, 2024

[Enhancement] active mv automatically (#32829)

488d937

Signed-off-by: Murphy <[email protected]> (cherry picked from commit c1e3a1e) Signed-off-by: Murphy <[email protected]>

murphyatwork added a commit that referenced this pull request Jan 5, 2024

[Enhancement] active mv automatically (#32829)

9f08603

Signed-off-by: Murphy <[email protected]> (cherry picked from commit c1e3a1e) Signed-off-by: Murphy <[email protected]>

wanpengfei-git pushed a commit that referenced this pull request Jan 6, 2024

[Enhancement] active mv automatically (backport #32829) (#38521)

1c4eb9a

Signed-off-by: Murphy <[email protected]> Co-authored-by: Murphy <[email protected]> Co-authored-by: Murphy <[email protected]>

github-actions bot added the 2.5-merged label Jan 6, 2024

wanpengfei-git pushed a commit that referenced this pull request Jan 7, 2024

[Enhancement] active mv automatically (backport #32829) (#32983)

f1b3217

Signed-off-by: Murphy <[email protected]> Co-authored-by: Murphy <[email protected]> Co-authored-by: Murphy <[email protected]>

github-actions bot added the 3.0-merged label Jan 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] active mv automatically #32829

[Enhancement] active mv automatically #32829

murphyatwork commented Oct 16, 2023 •

edited

Loading

starrocks-cr bot Oct 16, 2023

murphyatwork commented Oct 16, 2023

mergify bot commented Oct 16, 2023

ABingHuang Oct 17, 2023

murphyatwork Oct 17, 2023

ABingHuang commented Oct 17, 2023

sonarcloud bot commented Oct 17, 2023

wanpengfei-git commented Oct 17, 2023

wanpengfei-git commented Oct 17, 2023

wanpengfei-git commented Oct 17, 2023

wanpengfei-git commented Oct 17, 2023

mergify bot commented Oct 17, 2023 •

edited

Loading

wanpengfei-git commented Oct 17, 2023

mergify bot commented Oct 17, 2023 •

edited

Loading

mergify bot commented Oct 17, 2023 •

edited

Loading

murphyatwork commented Jan 5, 2024

mergify bot commented Jan 5, 2024 •

edited

Loading

[Enhancement] active mv automatically #32829

[Enhancement] active mv automatically #32829

Conversation

murphyatwork commented Oct 16, 2023 • edited Loading

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

starrocks-cr bot Oct 16, 2023

Choose a reason for hiding this comment

murphyatwork commented Oct 16, 2023

mergify bot commented Oct 16, 2023

✅ Branch has been successfully rebased

ABingHuang Oct 17, 2023

Choose a reason for hiding this comment

murphyatwork Oct 17, 2023

Choose a reason for hiding this comment

ABingHuang commented Oct 17, 2023

sonarcloud bot commented Oct 17, 2023

wanpengfei-git commented Oct 17, 2023

[BE Incremental Coverage Report]

wanpengfei-git commented Oct 17, 2023

[FE Incremental Coverage Report]

file detail

wanpengfei-git commented Oct 17, 2023

wanpengfei-git commented Oct 17, 2023

mergify bot commented Oct 17, 2023 • edited Loading

✅ Backports have been created

wanpengfei-git commented Oct 17, 2023

mergify bot commented Oct 17, 2023 • edited Loading

✅ Backports have been created

mergify bot commented Oct 17, 2023 • edited Loading

✅ Backports have been created

murphyatwork commented Jan 5, 2024

mergify bot commented Jan 5, 2024 • edited Loading

✅ Backports have been created

murphyatwork commented Oct 16, 2023 •

edited

Loading

mergify bot commented Oct 17, 2023 •

edited

Loading

mergify bot commented Oct 17, 2023 •

edited

Loading

mergify bot commented Oct 17, 2023 •

edited

Loading

mergify bot commented Jan 5, 2024 •

edited

Loading