[BugFix] Adding dynamic partition failure for one table may block other tables #51440
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
banmoy
force-pushed
the
fix_partition
branch
from
a266dc9
to
a4f3229
Compare
…ate dynamic partition Signed-off-by: PengFei Li <[email protected]>
banmoy
changed the title
banmoy
changed the title
|
[Java-Extensions Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[FE Incremental Coverage Report]✅ pass : 1 / 1 (100.00%) file detail
|
[BE Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
banmoy
changed the title
gengjun-git
approved these changes
Oct 9, 2024
HangyuanLiu
approved these changes
Oct 9, 2024
|
@Mergifyio backport branch-3.3 |
✅ Backports have been created
|
mergify bot
pushed a commit
that referenced
this pull request
Oct 9, 2024
…er tables (#51440) ## Why I'm doing: This bug is introduced by #47106. It changes [`DdlException` to `SemanticException`](https://github.com/StarRocks/starrocks/pull/47106/files#diff-d823bc00d682d8a36f2257a3ddcfa045cd2f248ef768d7f9fb0c9a53b039d82cR1035) which leads to `DynamicPartitionScheduler.executeDynamicPartitionForTable` can't catch the DdlException if adding partition fails. As a result, the current round of scheduling will stop, and other tables' partitions will not be created. A way to reproduce is as following. Create a partition named `p20241008` explicitly which will conflict with the dynamic partition to create. Dynamic partition will report exception `Getting analyzing error. Detail message: Duplicate partition name p20241008`. Note that change `p20241008` to the current day you execute the sql. ``` admin set frontend config ('dynamic_partition_check_interval_seconds' = '5'); admin set frontend config ('default_replication_num' = '1'); create database test; CREATE TABLE test.site_access1( event_day DATE, site_id INT DEFAULT '10', city_code VARCHAR(100), user_name VARCHAR(32) DEFAULT '', pv BIGINT DEFAULT '0' ) DUPLICATE KEY(event_day, site_id, city_code, user_name) PARTITION BY RANGE(event_day)( PARTITION p20241008 VALUES LESS THAN ("2020-03-25") ) DISTRIBUTED BY HASH(event_day, site_id) PROPERTIES( "dynamic_partition.enable" = "true", "dynamic_partition.time_unit" = "DAY", "dynamic_partition.end" = "3", "dynamic_partition.prefix" = "p", "dynamic_partition.history_partition_num" = "0" ); ``` ## What I'm doing: catch `Exception` rather than `DdlException` in `DynamicPartitionScheduler.executeDynamicPartitionForTable` Signed-off-by: PengFei Li <[email protected]> (cherry picked from commit ca8502a) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/clone/DynamicPartitionScheduler.java
42 tasks
banmoy
added a commit
to banmoy/starrocks
that referenced
this pull request
Oct 9, 2024
…er tables (StarRocks#51440) This bug is introduced by StarRocks#47106. It changes [`DdlException` to `SemanticException`](https://github.com/StarRocks/starrocks/pull/47106/files#diff-d823bc00d682d8a36f2257a3ddcfa045cd2f248ef768d7f9fb0c9a53b039d82cR1035) which leads to `DynamicPartitionScheduler.executeDynamicPartitionForTable` can't catch the DdlException if adding partition fails. As a result, the current round of scheduling will stop, and other tables' partitions will not be created. A way to reproduce is as following. Create a partition named `p20241008` explicitly which will conflict with the dynamic partition to create. Dynamic partition will report exception `Getting analyzing error. Detail message: Duplicate partition name p20241008`. Note that change `p20241008` to the current day you execute the sql. ``` admin set frontend config ('dynamic_partition_check_interval_seconds' = '5'); admin set frontend config ('default_replication_num' = '1'); create database test; CREATE TABLE test.site_access1( event_day DATE, site_id INT DEFAULT '10', city_code VARCHAR(100), user_name VARCHAR(32) DEFAULT '', pv BIGINT DEFAULT '0' ) DUPLICATE KEY(event_day, site_id, city_code, user_name) PARTITION BY RANGE(event_day)( PARTITION p20241008 VALUES LESS THAN ("2020-03-25") ) DISTRIBUTED BY HASH(event_day, site_id) PROPERTIES( "dynamic_partition.enable" = "true", "dynamic_partition.time_unit" = "DAY", "dynamic_partition.end" = "3", "dynamic_partition.prefix" = "p", "dynamic_partition.history_partition_num" = "0" ); ``` catch `Exception` rather than `DdlException` in `DynamicPartitionScheduler.executeDynamicPartitionForTable` Signed-off-by: PengFei Li <[email protected]>
24 tasks
gengjun-git
pushed a commit
that referenced
this pull request
Oct 9, 2024
…er tables (backport #51440) (#51667) backport #51440 Signed-off-by: PengFei Li <[email protected]>
ZiheLiu
pushed a commit
to ZiheLiu/starrocks
that referenced
this pull request
Oct 31, 2024
…er tables (StarRocks#51440) ## Why I'm doing: This bug is introduced by StarRocks#47106. It changes [`DdlException` to `SemanticException`](https://github.com/StarRocks/starrocks/pull/47106/files#diff-d823bc00d682d8a36f2257a3ddcfa045cd2f248ef768d7f9fb0c9a53b039d82cR1035) which leads to `DynamicPartitionScheduler.executeDynamicPartitionForTable` can't catch the DdlException if adding partition fails. As a result, the current round of scheduling will stop, and other tables' partitions will not be created. A way to reproduce is as following. Create a partition named `p20241008` explicitly which will conflict with the dynamic partition to create. Dynamic partition will report exception `Getting analyzing error. Detail message: Duplicate partition name p20241008`. Note that change `p20241008` to the current day you execute the sql. ``` admin set frontend config ('dynamic_partition_check_interval_seconds' = '5'); admin set frontend config ('default_replication_num' = '1'); create database test; CREATE TABLE test.site_access1( event_day DATE, site_id INT DEFAULT '10', city_code VARCHAR(100), user_name VARCHAR(32) DEFAULT '', pv BIGINT DEFAULT '0' ) DUPLICATE KEY(event_day, site_id, city_code, user_name) PARTITION BY RANGE(event_day)( PARTITION p20241008 VALUES LESS THAN ("2020-03-25") ) DISTRIBUTED BY HASH(event_day, site_id) PROPERTIES( "dynamic_partition.enable" = "true", "dynamic_partition.time_unit" = "DAY", "dynamic_partition.end" = "3", "dynamic_partition.prefix" = "p", "dynamic_partition.history_partition_num" = "0" ); ``` ## What I'm doing: catch `Exception` rather than `DdlException` in `DynamicPartitionScheduler.executeDynamicPartitionForTable` Signed-off-by: PengFei Li <[email protected]>
renzhimin7
pushed a commit
to renzhimin7/starrocks
that referenced
this pull request
Nov 7, 2024
…er tables (StarRocks#51440) ## Why I'm doing: This bug is introduced by StarRocks#47106. It changes [`DdlException` to `SemanticException`](https://github.com/StarRocks/starrocks/pull/47106/files#diff-d823bc00d682d8a36f2257a3ddcfa045cd2f248ef768d7f9fb0c9a53b039d82cR1035) which leads to `DynamicPartitionScheduler.executeDynamicPartitionForTable` can't catch the DdlException if adding partition fails. As a result, the current round of scheduling will stop, and other tables' partitions will not be created. A way to reproduce is as following. Create a partition named `p20241008` explicitly which will conflict with the dynamic partition to create. Dynamic partition will report exception `Getting analyzing error. Detail message: Duplicate partition name p20241008`. Note that change `p20241008` to the current day you execute the sql. ``` admin set frontend config ('dynamic_partition_check_interval_seconds' = '5'); admin set frontend config ('default_replication_num' = '1'); create database test; CREATE TABLE test.site_access1( event_day DATE, site_id INT DEFAULT '10', city_code VARCHAR(100), user_name VARCHAR(32) DEFAULT '', pv BIGINT DEFAULT '0' ) DUPLICATE KEY(event_day, site_id, city_code, user_name) PARTITION BY RANGE(event_day)( PARTITION p20241008 VALUES LESS THAN ("2020-03-25") ) DISTRIBUTED BY HASH(event_day, site_id) PROPERTIES( "dynamic_partition.enable" = "true", "dynamic_partition.time_unit" = "DAY", "dynamic_partition.end" = "3", "dynamic_partition.prefix" = "p", "dynamic_partition.history_partition_num" = "0" ); ``` ## What I'm doing: catch `Exception` rather than `DdlException` in `DynamicPartitionScheduler.executeDynamicPartitionForTable` Signed-off-by: PengFei Li <[email protected]> Signed-off-by: zhiminr.ren <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I'm doing:
This bug is introduced by #47106. It changes
DdlExceptiontoSemanticExceptionwhich leads toDynamicPartitionScheduler.executeDynamicPartitionForTablecan't catch the DdlException if adding partition fails. As a result, the current round of scheduling will stop, and other tables' partitions will not be created.A way to reproduce is as following. Create a partition named
p20241008explicitly which will conflict with the dynamic partition to create. Dynamic partition will report exceptionGetting analyzing error. Detail message: Duplicate partition name p20241008. Note that changep20241008to the current day you execute the sql.What I'm doing:
catch
Exceptionrather thanDdlExceptioninDynamicPartitionScheduler.executeDynamicPartitionForTableWhat type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: