Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] Partial update support const expr #50287

Merged
merged 9 commits into from
Sep 9, 2024

Conversation

sevev
Copy link
Contributor

@sevev sevev commented Aug 27, 2024

Why I'm doing:

Partial update does not support expression currently. If an expression column is not specified in the update, we will fill it with the default value instead of calculating it based on the expression.

What I'm doing:

Support partial update with expr and we only support default expr can be calculated like now() and does not support the expr for a batch of every row different like uuid(). If the default expr is uuid(), the behavior is not changed.
e.g.

 CREATE TABLE `test` (
  `k1` bigint(20) NOT NULL COMMENT "",
  `v1` bigint(20) NOT NULL COMMENT "",
  `v2` datetime NULL DEFAULT CURRENT_TIMESTAMP COMMENT ""
) ENGINE=OLAP
PRIMARY KEY(`k1`)
DISTRIBUTED BY HASH(`k1`) BUCKETS 1
PROPERTIES (
"compression" = "LZ4",
"enable_persistent_index" = "true",
"fast_schema_evolution" = "true",
"replicated_storage" = "true",
"replication_num" = "1"
);
insert into test values (1,1,'2024-08-27 00:00:00');
insert into test (k1,v1) values(2,2);
select * from test;

before this pr, the result is

mysql> select * from test;
+------+------+---------------------+
| k1   | v1   | v2                  |
+------+------+---------------------+
|    2 |    2 | NULL                |
|    1 |    1 | 2024-08-27 00:00:00 |
+------+------+---------------------+
2 rows in set (0.02 sec)

after this pr:

mysql> select * from test;
+------+------+---------------------+
| k1   | v1   | v2                  |
+------+------+---------------------+
|    1 |    1 | 2024-08-27 00:00:00 |
|    2 |    2 | 2024-08-27 19:02:23 |
+------+------+---------------------+
2 rows in set (0.01 sec)

Fixes #issue

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 3.3
    • 3.2
    • 3.1
    • 3.0
    • 2.5

@@ -675,7 +679,7 @@ Status RowsetUpdateState::_resolve_conflict_auto_increment(const RowsetUpdateSta
auto_increment_read_column.resize(1);
auto_increment_read_column[0] = _auto_increment_partial_update_states[segment_id].write_column->clone_empty();
RETURN_IF_ERROR(params.tablet->update_mgr()->get_column_values(
params, column_id, new_rows > 0, rowids_by_rssid, &auto_increment_read_column,
params, column_id, new_rows > 0, rowids_by_rssid, &auto_increment_read_column, &_column_to_expr_value,
&_auto_increment_partial_update_states[segment_id]));

std::unique_ptr<Column> new_write_column =
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The most risky bug in this code is:
Potentially missing initialization or improper handling of _column_to_expr_value

You can modify the code like this:

@@ -328,6 +328,7 @@ Status RowsetUpdateState::_prepare_auto_increment_partial_update_states(uint32_t
     }
 
+    _column_to_expr_value.clear();  // Ensure it's empty before usage
     RETURN_IF_ERROR(params.tablet->update_mgr()->get_column_values(params, column_id, new_rows > 0, rowids_by_rssid,
                                                                    &read_column, &_column_to_expr_value,
                                                                    &_auto_increment_partial_update_states[segment_id]));

@wanpengfei-git wanpengfei-git requested a review from a team August 27, 2024 02:15
@mergify mergify bot assigned sevev Aug 27, 2024
@sevev sevev force-pushed the partial_update_support_expr branch from dc7400c to 2447cfc Compare August 27, 2024 09:28
@sevev sevev changed the title [WIP][Enhancement] Partial update support expr [Enhancement] Partial update support expr Aug 28, 2024
Signed-off-by: sevev <[email protected]>
@sevev sevev force-pushed the partial_update_support_expr branch from f4bf8ed to 5f5a6fa Compare August 28, 2024 06:35
Signed-off-by: sevev <[email protected]>
Signed-off-by: sevev <[email protected]>
@sevev sevev changed the title [Enhancement] Partial update support expr [Enhancement] Partial update support const expr Sep 3, 2024
Signed-off-by: sevev <[email protected]>
decster
decster previously approved these changes Sep 4, 2024
Signed-off-by: sevev <[email protected]>
Copy link

sonarcloud bot commented Sep 6, 2024

Copy link

github-actions bot commented Sep 9, 2024

[Java-Extensions Incremental Coverage Report]

pass : 0 / 0 (0%)

Copy link

github-actions bot commented Sep 9, 2024

[FE Incremental Coverage Report]

pass : 4 / 4 (100.00%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 com/starrocks/planner/OlapTableSink.java 4 4 100.00% []

Copy link

github-actions bot commented Sep 9, 2024

[BE Incremental Coverage Report]

pass : 67 / 76 (88.16%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 be/src/runtime/lake_tablets_channel.cpp 3 5 60.00% [307, 310]
🔵 be/src/storage/lake/update_manager.cpp 6 10 60.00% [509, 510, 516, 517]
🔵 be/src/storage/lake/delta_writer.cpp 4 6 66.67% [502, 503]
🔵 be/src/storage/lake/rowset_update_state.cpp 4 5 80.00% [383]
🔵 be/src/storage/rowset_column_update_state.cpp 12 12 100.00% []
🔵 be/src/storage/lake/async_delta_writer.h 3 3 100.00% []
🔵 be/src/exec/tablet_info.cpp 7 7 100.00% []
🔵 be/src/storage/lake/delta_writer.h 3 3 100.00% []
🔵 be/src/storage/rowset/rowset_writer.cpp 3 3 100.00% []
🔵 be/src/storage/delta_writer.cpp 1 1 100.00% []
🔵 be/src/storage/rowset_update_state.cpp 4 4 100.00% []
🔵 be/src/runtime/local_tablets_channel.cpp 7 7 100.00% []
🔵 be/src/storage/tablet_updates.cpp 10 10 100.00% []

@sevev sevev merged commit 18ba78e into StarRocks:main Sep 9, 2024
47 of 50 checks passed
Copy link

github-actions bot commented Sep 9, 2024

@Mergifyio backport branch-3.3

@github-actions github-actions bot removed the 3.3 label Sep 9, 2024
Copy link

github-actions bot commented Sep 9, 2024

@Mergifyio backport branch-3.2

@github-actions github-actions bot removed the 3.2 label Sep 9, 2024
Copy link
Contributor

mergify bot commented Sep 9, 2024

backport branch-3.3

✅ Backports have been created

Copy link
Contributor

mergify bot commented Sep 9, 2024

backport branch-3.2

✅ Backports have been created

mergify bot pushed a commit that referenced this pull request Sep 9, 2024
mergify bot pushed a commit that referenced this pull request Sep 9, 2024
Signed-off-by: sevev <[email protected]>
(cherry picked from commit 18ba78e)

# Conflicts:
#	be/src/runtime/lake_tablets_channel.cpp
#	be/src/runtime/local_tablets_channel.h
#	be/src/storage/lake/async_delta_writer.cpp
#	be/src/storage/lake/async_delta_writer.h
#	be/src/storage/lake/delta_writer.cpp
#	be/src/storage/lake/delta_writer.h
#	be/src/storage/rowset/rowset_writer_context.h
#	fe/fe-core/src/main/java/com/starrocks/planner/OlapTableSink.java
HangyuanLiu pushed a commit to HangyuanLiu/starrocks that referenced this pull request Sep 12, 2024
wanpengfei-git pushed a commit that referenced this pull request Sep 27, 2024
renzhimin7 pushed a commit to renzhimin7/starrocks that referenced this pull request Nov 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants