[Enhancement] show persistent index disk cost in be_tablets #35615

luohaha · 2023-11-23T01:56:58Z

Why I'm doing:
We can't get the persistent index disk usage from system table now.

What I'm doing:
Add each tablet's persistent index disk usage INDEX_DISK to be_tablets.

mysql> select * from be_tablets where tablet_id = 10121\G
*************************** 1. row ***************************
       BE_ID: 10004
    TABLE_ID: 10119
PARTITION_ID: 10118
   TABLET_ID: 10121
 NUM_VERSION: 4
 MAX_VERSION: 6
 MIN_VERSION: 4
  NUM_ROWSET: 1
     NUM_ROW: 5
   DATA_SIZE: 710
   INDEX_MEM: 119
 CREATE_TIME: 1700701817
       STATE: RUNNING
        TYPE: PRIMARY
    DATA_DIR: /home/disk5/luoyixin/starrocks-4/starrocks/output/be/storage
    SHARD_ID: 48
 SCHEMA_HASH: 1872891624
  INDEX_DISK: 152
1 row in set (0.02 sec)

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

be/src/storage/tablet_updates.cpp

starrocks-cr · 2023-11-23T01:57:47Z

be/src/exec/schema_scanner/schema_be_tablets_scanner.cpp

+                // INDEX_DISK
+                fill_column_with_slot<TYPE_BIGINT>(column.get(), (void*)&info.index_disk_usage);
+                break;
+            }
            default:
                break;
            }


The most risky bug in this code is:
Incorrect memory size definition for TYPE_VARCHAR columns in the _s_columns[] array initialization.

You can modify the code like this:

@@ -34,7 +34,7 @@ SchemaScanner::ColumnDesc SchemaBeTabletsScanner::_s_columns[] = { {"INDEX_MEM", TYPE_BIGINT, sizeof(int64_t), false}, {"CREATE_TIME", TYPE_BIGINT, sizeof(int64_t), false}, // SIZE_VARCHAR has been changed from sizeof(StringValue) to an appropriate fixed buffer size or variable length handling mechanism {"STATE", TYPE_VARCHAR, appropriate_buffer_size, false}, {"TYPE", TYPE_VARCHAR, appropriate_buffer_size, false}, {"DATA_DIR", TYPE_VARCHAR, appropriate_buffer_size, false}, {"SHARD_ID", TYPE_BIGINT, sizeof(int64_t), false}, - {"SCHEMA_HASH", TYPE_BIGINT, sizeof(int64_t), false}, + {"SCHEMA_HASH", TYPE_BIGINT, sizeof(int64_t), false}, {"INDEX_DISK", TYPE_BIGINT, sizeof(int64_t), false}, };

Remember to replace appropriate_buffer_size with the actual fixed size or dynamic size handling logic based on the real use case and data requirements.

Explanation:
The TYPE_VARCHAR likely expects a string value and therefore needs a correct buffer length to accommodate the string data. The sizeof(StringValue) here might not reflect the actual size needed for the VARCHAR data depending on how StringValue is defined. In common scenarios, VARCHAR types either specify a maximum length or are accompanied by dynamic sizing logic in the system to handle variable-length strings appropriately.

be/src/storage/tablet_updates.cpp

Signed-off-by: luohaha <[email protected]>

sonarcloud · 2023-11-28T22:38:10Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

0.0% Coverage
0.0% Duplication

The version of Java (11.0.21) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more here

github-actions · 2023-11-28T23:59:32Z

[FE Incremental Coverage Report]

✅ pass : 1 / 1 (100.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/catalog/system/information/BeTabletsSystemTable.java	1	1	100.00%	[]

github-actions · 2023-11-29T00:00:26Z

[BE Incremental Coverage Report]

❌ fail : 0 / 14 (00.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	src/storage/tablet_updates.cpp	0	10	00.00%	[2823, 2824, 2828, 2829, 3969, 3970, 3971, 3974, 3975, 3977]
🔵	src/exec/schema_scanner/schema_be_tablets_scanner.cpp	0	4	00.00%	[95, 188, 190, 191]

github-actions · 2023-11-29T02:09:26Z

@Mergifyio backport branch-3.2

github-actions · 2023-11-29T02:09:28Z

@Mergifyio backport branch-3.1

mergify · 2023-11-29T02:09:30Z

backport branch-3.2

✅ Backports have been created

#36022 [Enhancement] show persistent index disk cost in be_tablets (backport #35615) has been created for branch branch-3.2

mergify · 2023-11-29T02:09:32Z

backport branch-3.1

✅ Backports have been created

#36023 [Enhancement] show persistent index disk cost in be_tablets (backport #35615) has been created for branch branch-3.1 but encountered conflicts

We can't get the persistent index disk usage from system table now. Add each tablet's persistent index disk usage `INDEX_DISK` to `be_tablets`. ``` mysql> select * from be_tablets where tablet_id = 10121\G *************************** 1. row *************************** BE_ID: 10004 TABLE_ID: 10119 PARTITION_ID: 10118 TABLET_ID: 10121 NUM_VERSION: 4 MAX_VERSION: 6 MIN_VERSION: 4 NUM_ROWSET: 1 NUM_ROW: 5 DATA_SIZE: 710 INDEX_MEM: 119 CREATE_TIME: 1700701817 STATE: RUNNING TYPE: PRIMARY DATA_DIR: /home/disk5/luoyixin/starrocks-4/starrocks/output/be/storage SHARD_ID: 48 SCHEMA_HASH: 1872891624 INDEX_DISK: 152 1 row in set (0.02 sec) ``` (cherry picked from commit 956f397)

We can't get the persistent index disk usage from system table now. Add each tablet's persistent index disk usage `INDEX_DISK` to `be_tablets`. ``` mysql> select * from be_tablets where tablet_id = 10121\G *************************** 1. row *************************** BE_ID: 10004 TABLE_ID: 10119 PARTITION_ID: 10118 TABLET_ID: 10121 NUM_VERSION: 4 MAX_VERSION: 6 MIN_VERSION: 4 NUM_ROWSET: 1 NUM_ROW: 5 DATA_SIZE: 710 INDEX_MEM: 119 CREATE_TIME: 1700701817 STATE: RUNNING TYPE: PRIMARY DATA_DIR: /home/disk5/luoyixin/starrocks-4/starrocks/output/be/storage SHARD_ID: 48 SCHEMA_HASH: 1872891624 INDEX_DISK: 152 1 row in set (0.02 sec) ``` (cherry picked from commit 956f397) # Conflicts: # be/src/storage/tablet_updates.cpp # fe/fe-core/src/test/resources/sql/scheduler/schema_scan.sql

…35615) (#36022) Co-authored-by: Yixin Luo <[email protected]>

…s#35615) We can't get the persistent index disk usage from system table now. Add each tablet's persistent index disk usage `INDEX_DISK` to `be_tablets`. ``` mysql> select * from be_tablets where tablet_id = 10121\G *************************** 1. row *************************** BE_ID: 10004 TABLE_ID: 10119 PARTITION_ID: 10118 TABLET_ID: 10121 NUM_VERSION: 4 MAX_VERSION: 6 MIN_VERSION: 4 NUM_ROWSET: 1 NUM_ROW: 5 DATA_SIZE: 710 INDEX_MEM: 119 CREATE_TIME: 1700701817 STATE: RUNNING TYPE: PRIMARY DATA_DIR: /home/disk5/luoyixin/starrocks-4/starrocks/output/be/storage SHARD_ID: 48 SCHEMA_HASH: 1872891624 INDEX_DISK: 152 1 row in set (0.02 sec) ```

…35615) (#36280) Signed-off-by: luohaha <[email protected]>

…35615) (#36279)

github-actions bot added behavior_changed 3.2 3.1 labels Nov 23, 2023

starrocks-cr bot reviewed Nov 23, 2023

View reviewed changes

be/src/storage/tablet_updates.cpp Show resolved Hide resolved

starrocks-cr bot reviewed Nov 23, 2023

View reviewed changes

mergify bot assigned luohaha Nov 23, 2023

luohaha force-pushed the add-index-usage-to-be-tablet branch from 07153c2 to 5422d17 Compare November 27, 2023 18:59

chaoyli reviewed Nov 27, 2023

View reviewed changes

be/src/storage/tablet_updates.cpp Outdated Show resolved Hide resolved

[Enhancement] show persistent index disk cost in be_tablets

fe9626d

Signed-off-by: luohaha <[email protected]>

luohaha force-pushed the add-index-usage-to-be-tablet branch from 5422d17 to fe9626d Compare November 28, 2023 22:27

chaoyli approved these changes Nov 29, 2023

View reviewed changes

sevev approved these changes Nov 29, 2023

View reviewed changes

chaoyli merged commit 956f397 into StarRocks:main Nov 29, 2023
48 of 49 checks passed

github-actions bot removed the 3.2 label Nov 29, 2023

github-actions bot removed the 3.1 label Nov 29, 2023

This was referenced Nov 29, 2023

[Enhancement] show persistent index disk cost in be_tablets (backport #35615) #36022

Merged

[Enhancement] show persistent index disk cost in be_tablets (backport #35615) #36023

Closed

chaoyli pushed a commit that referenced this pull request Nov 30, 2023

[Enhancement] show persistent index disk cost in be_tablets (backport #…

37f1f80

…35615) (#36022) Co-authored-by: Yixin Luo <[email protected]>

luohaha mentioned this pull request Dec 1, 2023

[Enhancement] show persistent index disk cost in be_tablets (backport #35615) #36279

Merged

22 tasks

luohaha mentioned this pull request Dec 1, 2023

[Enhancement] show persistent index disk cost in be_tablets (backport #35615) #36280

Merged

22 tasks

chaoyli pushed a commit that referenced this pull request Dec 2, 2023

[Enhancement] show persistent index disk cost in be_tablets (backport #…

f13de63

…35615) (#36280) Signed-off-by: luohaha <[email protected]>

chaoyli pushed a commit that referenced this pull request Dec 2, 2023

[Enhancement] show persistent index disk cost in be_tablets (backport #…

618c032

…35615) (#36279)

wanpengfei-git added the backport_ok label Dec 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] show persistent index disk cost in be_tablets #35615

[Enhancement] show persistent index disk cost in be_tablets #35615

luohaha commented Nov 23, 2023 •

edited

Loading

starrocks-cr bot Nov 23, 2023

sonarcloud bot commented Nov 28, 2023

github-actions bot commented Nov 28, 2023

github-actions bot commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

mergify bot commented Nov 29, 2023 •

edited

Loading

mergify bot commented Nov 29, 2023 •

edited

Loading

[Enhancement] show persistent index disk cost in be_tablets #35615

[Enhancement] show persistent index disk cost in be_tablets #35615

Conversation

luohaha commented Nov 23, 2023 • edited Loading

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

starrocks-cr bot Nov 23, 2023

Choose a reason for hiding this comment

sonarcloud bot commented Nov 28, 2023

github-actions bot commented Nov 28, 2023

[FE Incremental Coverage Report]

file detail

github-actions bot commented Nov 29, 2023

[BE Incremental Coverage Report]

file detail

github-actions bot commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

mergify bot commented Nov 29, 2023 • edited Loading

✅ Backports have been created

mergify bot commented Nov 29, 2023 • edited Loading

✅ Backports have been created

luohaha commented Nov 23, 2023 •

edited

Loading

mergify bot commented Nov 29, 2023 •

edited

Loading

mergify bot commented Nov 29, 2023 •

edited

Loading