[Enhancement] Support profile for only big query (2) #39855

liuyehcf · 2024-01-24T06:20:02Z

Why I'm doing:

In the previous pr #33825, we've supported the profile only for big query with the session variable big_query_profile_second_threshold

Unfortunately, for some of our uses, most of the queries can be done within 200ms with a latency spike over 400ms, we cannot catch the slow query(400ms) with the session variable mentioned above.

What I'm doing & Behavior Change

Remove session variable big_query_profile_second_threshold with type int.
Add session variable big_query_profile_threshold with type string, it can support time duration like 1s/500ms/15m etc.

This change won't lead to compatible problem when performing grey upgrading.

Only upgrade some of the backends, it works fine with the previous session variable big_query_profile_second_threshold.
Only upgrade all of the backends, it works fine with the previous session variable big_query_profile_second_threshold
And after upgrading the frontend, the user need to set up the big_query_profile_threshold corresponding to the previous configuration manually.

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

Signed-off-by: liuyehcf <[email protected]>

fe/fe-core/src/main/java/com/starrocks/qe/SessionVariable.java

Signed-off-by: liuyehcf <[email protected]>

allencelerdata · 2024-01-24T23:43:04Z

fe/fe-core/src/main/java/com/starrocks/qe/SessionVariable.java

+    @VariableMgr.VarAttr(name = BIG_QUERY_PROFILE_THRESHOLD, alias = BIG_QUERY_PROFILE_SECOND_THRESHOLD)
+    private long bigQueryProfileThreshold = 0;
+
+    @VariableMgr.VarAttr(name = BIG_QUERY_PROFILE_THRESHOLD_UNIT)


No need to add a new big_query_profile_threshold_unit. You can use a string like 1s/1ms/1min and we parse the value and convert to the actual int value

Signed-off-by: liuyehcf <[email protected]>

docs/en/administration/query_profile_overview.md

Signed-off-by: liuyehcf <[email protected]>

packy92 · 2024-01-26T02:26:14Z

docs/zh/administration/query_profile_overview.md

@@ -16,10 +16,17 @@ SET enable_profile = true;

 ### 针对慢查询开启 Query Profile

-在生产环境中，通常不推荐全面启用 Query Profile 功能。这是因为 Query Profile 的数据采集和处理过程可能会为系统带来额外的负担。然而，如果需要捕捉到耗时的慢查询，就需要巧妙地使用这一功能。为此，您可以选择只对慢查询启用 Query Profile。这可以通过设置变量 `big_query_profile_second_threshold` 为一个大于 `0` 的整数来实现。例如，若将此变量设置为 30，意味着只有那些执行时间超过 30 秒的查询会启用 Query Profile 功能。这样既保证了系统性能，又能有效监控到慢查询。
+在生产环境中，通常不推荐全面启用 Query Profile 功能。这是因为 Query Profile 的数据采集和处理过程可能会为系统带来额外的负担。然而，如果需要捕捉到耗时的慢查询，就需要巧妙地使用这一功能。为此，您可以选择只对慢查询启用 Query Profile。这可以通过设置变量 `big_query_profile_threshold` 为一个大于 `0s` 的时间来实现。例如，若将此变量设置为 `30s`，意味着只有那些执行时间超过 30 秒的查询会启用 Query Profile 功能。这样既保证了系统性能，又能有效监控到慢查询。



Now, is the unit necessary when setting this?

yes, the type of this variable is string now, a time unit is mandatory.

sonarcloud · 2024-01-26T02:26:59Z

Quality Gate passed

The SonarCloud Quality Gate passed, but some issues were introduced.

1 New issue
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-01-26T04:18:33Z

[FE Incremental Coverage Report]

❌ fail : 15 / 34 (44.12%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/qe/ConnectContext.java	0	1	00.00%	[597]
🔵	com/starrocks/sql/analyzer/SetStmtAnalyzer.java	1	5	20.00%	[257, 258, 259, 260]
🔵	com/starrocks/monitor/unit/TimeValue.java	10	23	43.48%	[248, 250, 258, 259, 260, 261, 262, 263, 264, 265, 267, 269, 270]
🔵	com/starrocks/qe/SessionVariable.java	4	5	80.00%	[1933]

github-actions · 2024-01-26T04:26:10Z

[BE Incremental Coverage Report]

❌ fail : 8 / 24 (33.33%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	be/src/exec/pipeline/query_context.h	5	21	23.81%	[105, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123]
🔵	be/src/exec/pipeline/fragment_executor.cpp	3	3	100.00%	[]

github-actions · 2024-01-29T03:22:54Z

@Mergifyio backport branch-3.2

github-actions · 2024-01-29T03:22:56Z

@Mergifyio backport branch-3.1

mergify · 2024-01-29T03:22:57Z

backport branch-3.2

✅ Backports have been created

#40206 [Enhancement] Support profile for only big query (2) (backport #39855) has been created for branch branch-3.2 but encountered conflicts

mergify · 2024-01-29T03:22:58Z

backport branch-3.1

✅ Backports have been created

#40207 [Enhancement] Support profile for only big query (2) (backport #39855) has been created for branch branch-3.1 but encountered conflicts

Signed-off-by: liuyehcf <[email protected]> (cherry picked from commit a3758df) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/sql/analyzer/SetStmtAnalyzer.java # gensrc/thrift/InternalService.thrift

Signed-off-by: liuyehcf <[email protected]>

#40210) Signed-off-by: liuyehcf <[email protected]> Signed-off-by: zihe.liu <[email protected]> Co-authored-by: liuyehcf <[email protected]>

#40209) Signed-off-by: liuyehcf <[email protected]> Signed-off-by: zihe.liu <[email protected]> Co-authored-by: liuyehcf <[email protected]>

Signed-off-by: liuyehcf <[email protected]>

Signed-off-by: liuyehcf <[email protected]> Signed-off-by: Seaven <[email protected]>

[Enhancement] Big query profile support milliseconds threshold

0691f27

Signed-off-by: liuyehcf <[email protected]>

liuyehcf requested a review from a team as a code owner January 24, 2024 06:20

github-actions bot added the PROTO-REVIEW label Jan 24, 2024

mergify bot assigned liuyehcf Jan 24, 2024

github-actions bot added 3.2 3.1 labels Jan 24, 2024

allencelerdata reviewed Jan 24, 2024

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/qe/SessionVariable.java Outdated Show resolved Hide resolved

liuyehcf force-pushed the profile2 branch from 7e6ee25 to da6ea5c Compare January 24, 2024 11:14

update 1: add unit

c999b9f

Signed-off-by: liuyehcf <[email protected]>

liuyehcf force-pushed the profile2 branch from da6ea5c to c999b9f Compare January 24, 2024 11:17

allencelerdata reviewed Jan 24, 2024

View reviewed changes

liuyehcf added 2 commits January 25, 2024 09:41

update2: remove unit variable

bf23ae4

Signed-off-by: liuyehcf <[email protected]>

update3: doc

068e720

Signed-off-by: liuyehcf <[email protected]>

github-actions bot added documentation Improvements or additions to documentation behavior_changed labels Jan 25, 2024

update 4: doc

0a4a310

Signed-off-by: liuyehcf <[email protected]>

github-actions bot removed the behavior_changed label Jan 25, 2024

update 4: use TimeValue to parse config

8e48617

Signed-off-by: liuyehcf <[email protected]>

liuyehcf requested a review from a team as a code owner January 25, 2024 08:02

update 5: change parse logic

776a18e

Signed-off-by: liuyehcf <[email protected]>

chaoyli changed the title ~~[Enhancement] Support profile for only big query (2)~~ [Enhancement] Support profile for only big query Jan 25, 2024

chaoyli previously approved these changes Jan 26, 2024

View reviewed changes

kangkaisen reviewed Jan 26, 2024

View reviewed changes

docs/en/administration/query_profile_overview.md Outdated Show resolved Hide resolved

update 6: fix doc

27b0ab9

Signed-off-by: liuyehcf <[email protected]>

liuyehcf dismissed chaoyli’s stale review via 27b0ab9 January 26, 2024 02:19

packy92 approved these changes Jan 26, 2024

View reviewed changes

liuyehcf changed the title ~~[Enhancement] Support profile for only big query~~ [Enhancement] Support profile for only big query (2) Jan 26, 2024

kangkaisen approved these changes Jan 26, 2024

View reviewed changes

Seaven approved these changes Jan 26, 2024

View reviewed changes

Seaven enabled auto-merge (squash) January 26, 2024 06:56

kangkaisen disabled auto-merge January 29, 2024 03:22

kangkaisen merged commit a3758df into StarRocks:main Jan 29, 2024
58 of 61 checks passed

github-actions bot removed the 3.2 label Jan 29, 2024

github-actions bot removed the 3.1 label Jan 29, 2024

mergify bot mentioned this pull request Jan 29, 2024

[Enhancement] Support profile for only big query (2) (backport #39855) #40206

Closed

18 tasks

mergify bot mentioned this pull request Jan 29, 2024

[Enhancement] Support profile for only big query (2) (backport #39855) #40207

Closed

18 tasks

ZiheLiu pushed a commit to ZiheLiu/starrocks that referenced this pull request Jan 29, 2024

[Enhancement] Support profile for only big query (2) (StarRocks#39855)

87740fb

Signed-off-by: liuyehcf <[email protected]>

This was referenced Jan 29, 2024

[Enhancement] Support profile for only big query (2) (backport #39855) #40209

Merged

[Enhancement] Support profile for only big query (2) (backport #39855) #40210

Merged

ZiheLiu pushed a commit to ZiheLiu/starrocks that referenced this pull request Jan 29, 2024

[Enhancement] Support profile for only big query (2) (StarRocks#39855)

f144acb

Signed-off-by: liuyehcf <[email protected]>

chaoyli pushed a commit to chaoyli/starrocks that referenced this pull request Jan 29, 2024

[Enhancement] Support profile for only big query (2) (StarRocks#39855)

a4a7b60

Signed-off-by: liuyehcf <[email protected]>

github-actions bot added 3.1-merged 3.2-merged labels Jan 29, 2024

motto1314 pushed a commit to motto1314/starrocks that referenced this pull request Jan 30, 2024

[Enhancement] Support profile for only big query (2) (StarRocks#39855)

f3f1d32

Signed-off-by: liuyehcf <[email protected]>

Seaven pushed a commit to Seaven/starrocks that referenced this pull request Jan 30, 2024

[Enhancement] Support profile for only big query (2) (StarRocks#39855)

9579efc

Signed-off-by: liuyehcf <[email protected]> Signed-off-by: Seaven <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Support profile for only big query (2) #39855

[Enhancement] Support profile for only big query (2) #39855

liuyehcf commented Jan 24, 2024 •

edited by github-actions bot

Loading

allencelerdata Jan 24, 2024

liuyehcf Jan 25, 2024

packy92 Jan 26, 2024

liuyehcf Jan 26, 2024

sonarcloud bot commented Jan 26, 2024

github-actions bot commented Jan 26, 2024

github-actions bot commented Jan 26, 2024

github-actions bot commented Jan 29, 2024

github-actions bot commented Jan 29, 2024

mergify bot commented Jan 29, 2024 •

edited

Loading

mergify bot commented Jan 29, 2024 •

edited

Loading

[Enhancement] Support profile for only big query (2) #39855

[Enhancement] Support profile for only big query (2) #39855

Conversation

liuyehcf commented Jan 24, 2024 • edited by github-actions bot Loading

Why I'm doing:

What I'm doing & Behavior Change

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

allencelerdata Jan 24, 2024

Choose a reason for hiding this comment

liuyehcf Jan 25, 2024

Choose a reason for hiding this comment

packy92 Jan 26, 2024

Choose a reason for hiding this comment

liuyehcf Jan 26, 2024

Choose a reason for hiding this comment

sonarcloud bot commented Jan 26, 2024

Quality Gate passed

github-actions bot commented Jan 26, 2024

[FE Incremental Coverage Report]

file detail

github-actions bot commented Jan 26, 2024

[BE Incremental Coverage Report]

file detail

github-actions bot commented Jan 29, 2024

github-actions bot commented Jan 29, 2024

mergify bot commented Jan 29, 2024 • edited Loading

✅ Backports have been created

mergify bot commented Jan 29, 2024 • edited Loading

✅ Backports have been created

liuyehcf commented Jan 24, 2024 •

edited by github-actions bot

Loading

mergify bot commented Jan 29, 2024 •

edited

Loading

mergify bot commented Jan 29, 2024 •

edited

Loading