[Enhancement] Support profile for only big query #33825

liuyehcf · 2023-10-27T07:03:51Z

Background

In practical system, we won't enable the profile, because this feature may have significant overhead and may lead to decrease in performance. But some slow query may occur on a regular basis, and we want to get the profile of those queries, which is not supported in our system currently.

How

In this pr, we offer an additional session variable, namely big_query_profile_second_threshold, to enable only the slow query.

And here's how it works:

if enable_profile=true, than the added variable is simply ignored.
If enable_profile=false and big_query_profile_second_threshold > 0. Backend will process profile and report profile if overall time exceed the threshold(defined by big_query_profile_second_threshold). And frontend will finally decide whether the profile will be displayed or not based on the comparation between the overall query time and big_query_profile_second_threshold, if the overall query time is greater then the value of big_query_profile_second_threshold, the profile can be seen, otherwise not.

Limitation

A large query might consist of several fragment instances. Some of these instances finish quickly after execution starts, while others may take several minutes. Only those fragment instances that last longer than the bigQueryProfileSecondThreshold will report their profiles. This results in an incomplete profile that includes only the most time-consuming parts

Example

https://gist.github.com/liuyehcf/a4ec577d188622cf16d324d0491e6e69

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

be/src/exec/pipeline/fragment_executor.cpp

Signed-off-by: liuyehcf <[email protected]>

sonarcloud · 2023-10-30T09:27:25Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
7 Code Smells

0.0% Coverage
0.0% Duplication

github-actions · 2023-10-30T11:27:28Z

[FE Incremental Coverage Report]

❌ fail : 21 / 30 (70.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/load/loadv2/LoadLoadingTask.java	0	1	00.00%	[220]
🔵	com/starrocks/load/streamload/StreamLoadTask.java	0	1	00.00%	[1061]
🔵	com/starrocks/qe/StmtExecutor.java	2	4	50.00%	[793, 802]
🔵	com/starrocks/qe/scheduler/QueryRuntimeProfile.java	2	3	66.67%	[277]
🔵	com/starrocks/qe/ConnectContext.java	7	10	70.00%	[563, 571, 572]
🔵	com/starrocks/qe/SessionVariable.java	3	4	75.00%	[1665]
🔵	com/starrocks/common/profile/Tracers.java	2	2	100.00%	[]
🔵	com/starrocks/qe/scheduler/dag/JobSpec.java	2	2	100.00%	[]
🔵	com/starrocks/qe/DefaultCoordinator.java	3	3	100.00%	[]

github-actions · 2023-10-30T11:28:47Z

[BE Incremental Coverage Report]

❌ fail : 4 / 10 (40.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	src/exec/pipeline/fragment_executor.cpp	0	2	00.00%	[138, 139]
🔵	src/exec/pipeline/query_context.h	4	8	50.00%	[94, 99, 101, 102]

github-actions · 2023-10-30T11:38:45Z

@Mergifyio backport branch-3.2

github-actions · 2023-10-30T11:38:47Z

@Mergifyio backport branch-3.1

mergify · 2023-10-30T11:38:48Z

backport branch-3.2

✅ Backports have been created

#33948 [Enhancement] Support profile for only big query (backport #33825) has been created for branch branch-3.2 but encountered conflicts

Signed-off-by: liuyehcf <[email protected]> (cherry picked from commit 0e2d056) # Conflicts: # gensrc/thrift/InternalService.thrift

mergify · 2023-10-30T11:39:15Z

backport branch-3.1

✅ Backports have been created

#33949 [Enhancement] Support profile for only big query (backport #33825) has been created for branch branch-3.1 but encountered conflicts

Signed-off-by: liuyehcf <[email protected]> (cherry picked from commit 0e2d056) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/common/profile/Tracers.java # fe/fe-core/src/main/java/com/starrocks/qe/DefaultCoordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/Coordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/QueryRuntimeProfile.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/dag/JobSpec.java # gensrc/thrift/InternalService.thrift

Signed-off-by: liuyehcf <[email protected]> (cherry picked from commit 0e2d056) # Conflicts: # gensrc/thrift/InternalService.thrift

Signed-off-by: liuyehcf <[email protected]> (cherry picked from commit 0e2d056) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/common/profile/Tracers.java # fe/fe-core/src/main/java/com/starrocks/qe/DefaultCoordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/Coordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/QueryRuntimeProfile.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/dag/JobSpec.java # gensrc/thrift/InternalService.thrift

liuyehcf · 2023-10-31T06:21:05Z

@Mergifyio backport branch-2.5

mergify · 2023-10-31T06:21:10Z

backport branch-2.5

✅ Backports have been created

#34016 [Enhancement] Support profile for only big query (backport #33825) has been created for branch branch-2.5 but encountered conflicts

liuyehcf · 2023-10-31T06:21:13Z

@Mergifyio backport branch-3.0

mergify · 2023-10-31T06:21:19Z

backport branch-3.0

✅ Backports have been created

#34017 [Enhancement] Support profile for only big query (backport #33825) has been created for branch branch-3.0 but encountered conflicts

Signed-off-by: liuyehcf <[email protected]> (cherry picked from commit 0e2d056) # Conflicts: # be/src/exec/pipeline/query_context.h # fe/fe-core/src/main/java/com/starrocks/common/profile/Tracers.java # fe/fe-core/src/main/java/com/starrocks/load/loadv2/LoadLoadingTask.java # fe/fe-core/src/main/java/com/starrocks/load/streamload/StreamLoadTask.java # fe/fe-core/src/main/java/com/starrocks/qe/ConnectContext.java # fe/fe-core/src/main/java/com/starrocks/qe/DefaultCoordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/SessionVariable.java # fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/Coordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/QueryRuntimeProfile.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/dag/JobSpec.java # gensrc/thrift/InternalService.thrift

Signed-off-by: liuyehcf <[email protected]> (cherry picked from commit 0e2d056) # Conflicts: # be/src/exec/pipeline/query_context.h # fe/fe-core/src/main/java/com/starrocks/common/profile/Tracers.java # fe/fe-core/src/main/java/com/starrocks/load/loadv2/LoadLoadingTask.java # fe/fe-core/src/main/java/com/starrocks/load/streamload/StreamLoadTask.java # fe/fe-core/src/main/java/com/starrocks/qe/DefaultCoordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/SessionVariable.java # fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/Coordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/QueryRuntimeProfile.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/dag/JobSpec.java # gensrc/thrift/InternalService.thrift

Signed-off-by: liuyehcf <[email protected]> (cherry picked from commit 0e2d056) # Conflicts: # be/src/exec/pipeline/query_context.h # fe/fe-core/src/main/java/com/starrocks/common/profile/Tracers.java # fe/fe-core/src/main/java/com/starrocks/load/loadv2/LoadLoadingTask.java # fe/fe-core/src/main/java/com/starrocks/load/streamload/StreamLoadTask.java # fe/fe-core/src/main/java/com/starrocks/qe/ConnectContext.java # fe/fe-core/src/main/java/com/starrocks/qe/DefaultCoordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/SessionVariable.java # fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/Coordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/QueryRuntimeProfile.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/dag/JobSpec.java # gensrc/thrift/InternalService.thrift

Signed-off-by: liuyehcf <[email protected]> (cherry picked from commit 0e2d056) # Conflicts: # be/src/exec/pipeline/query_context.h # fe/fe-core/src/main/java/com/starrocks/common/profile/Tracers.java # fe/fe-core/src/main/java/com/starrocks/load/loadv2/LoadLoadingTask.java # fe/fe-core/src/main/java/com/starrocks/load/streamload/StreamLoadTask.java # fe/fe-core/src/main/java/com/starrocks/qe/DefaultCoordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/SessionVariable.java # fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/Coordinator.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/QueryRuntimeProfile.java # fe/fe-core/src/main/java/com/starrocks/qe/scheduler/dag/JobSpec.java # gensrc/thrift/InternalService.thrift

Signed-off-by: liuyehcf <[email protected]> (cherry picked from commit 0e2d056)

liuyehcf · 2023-12-14T11:03:13Z

version >=3.0.8 or >=3.1.5

github-actions bot added PROTO-REVIEW 3.2 3.1 labels Oct 27, 2023

mergify bot assigned liuyehcf Oct 27, 2023

liuyehcf force-pushed the profile branch from aed6a99 to 329b574 Compare October 27, 2023 08:21

github-actions bot removed the PROTO-REVIEW label Oct 27, 2023

liuyehcf changed the title ~~[Enhancement] Support profile only for long-running profile~~ [Enhancement] Support profile for only big query Oct 30, 2023

liuyehcf force-pushed the profile branch from c9dc0c8 to 8057ffc Compare October 30, 2023 07:22

github-actions bot added the PROTO-REVIEW label Oct 30, 2023

kangkaisen reviewed Oct 30, 2023

View reviewed changes

be/src/exec/pipeline/fragment_executor.cpp Show resolved Hide resolved

[Enhancement] Support profile for only big query

28731e6

Signed-off-by: liuyehcf <[email protected]>

liuyehcf force-pushed the profile branch 2 times, most recently from 7dd9145 to 7c81810 Compare October 30, 2023 09:08

update 1

fe44bf0

Signed-off-by: liuyehcf <[email protected]>

liuyehcf force-pushed the profile branch from 7c81810 to fe44bf0 Compare October 30, 2023 09:10

kangkaisen approved these changes Oct 30, 2023

View reviewed changes

Seaven approved these changes Oct 30, 2023

View reviewed changes

liuyehcf enabled auto-merge (squash) October 30, 2023 11:34

kangkaisen disabled auto-merge October 30, 2023 11:38

kangkaisen merged commit 0e2d056 into StarRocks:main Oct 30, 2023
50 of 52 checks passed

github-actions bot removed the 3.2 label Oct 30, 2023

github-actions bot removed the 3.1 label Oct 30, 2023

mergify bot pushed a commit that referenced this pull request Oct 30, 2023

[Enhancement] Support profile for only big query (#33825)

858b6e7

Signed-off-by: liuyehcf <[email protected]> (cherry picked from commit 0e2d056) # Conflicts: # gensrc/thrift/InternalService.thrift

mergify bot mentioned this pull request Oct 30, 2023

[Enhancement] Support profile for only big query (backport #33825) #33948

Merged

mergify bot mentioned this pull request Oct 30, 2023

[Enhancement] Support profile for only big query (backport #33825) #33949

Merged

wanpengfei-git pushed a commit that referenced this pull request Oct 30, 2023

[Enhancement] Support profile for only big query (#33825)

b539a5d

Signed-off-by: liuyehcf <[email protected]> (cherry picked from commit 0e2d056) # Conflicts: # gensrc/thrift/InternalService.thrift

mergify bot mentioned this pull request Oct 31, 2023

[Enhancement] Support profile for only big query (backport #33825) #34016

Merged

mergify bot mentioned this pull request Oct 31, 2023

[Enhancement] Support profile for only big query (backport #33825) #34017

Merged

liuyehcf added a commit to liuyehcf/starrocks that referenced this pull request Nov 1, 2023

[Enhancement] Support profile for only big query (StarRocks#33825)

e73207f

Signed-off-by: liuyehcf <[email protected]> (cherry picked from commit 0e2d056)

liuyehcf added a commit to liuyehcf/starrocks that referenced this pull request Nov 1, 2023

[Enhancement] Support profile for only big query (StarRocks#33825)

31f49ca

Signed-off-by: liuyehcf <[email protected]> (cherry picked from commit 0e2d056)

github-actions bot added 3.2 3.1 labels Nov 29, 2023

wanpengfei-git added the backport_ok label Dec 5, 2023

liuyehcf mentioned this pull request Jan 24, 2024

[Enhancement] Support profile for only big query (2) #39855

Merged

23 tasks

This was referenced Jan 29, 2024

[Enhancement] Support profile for only big query (2) (backport #39855) #40206

Closed

[Enhancement] Support profile for only big query (2) (backport #39855) #40207

Closed

This was referenced Jan 29, 2024

[Enhancement] Support profile for only big query (2) (backport #39855) #40209

Merged

[Enhancement] Support profile for only big query (2) (backport #39855) #40210

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Support profile for only big query #33825

[Enhancement] Support profile for only big query #33825

liuyehcf commented Oct 27, 2023 •

edited

Loading

sonarcloud bot commented Oct 30, 2023

github-actions bot commented Oct 30, 2023

github-actions bot commented Oct 30, 2023

github-actions bot commented Oct 30, 2023

github-actions bot commented Oct 30, 2023

mergify bot commented Oct 30, 2023 •

edited

Loading

mergify bot commented Oct 30, 2023 •

edited

Loading

liuyehcf commented Oct 31, 2023

mergify bot commented Oct 31, 2023 •

edited

Loading

liuyehcf commented Oct 31, 2023

mergify bot commented Oct 31, 2023 •

edited

Loading

liuyehcf commented Dec 14, 2023 •

edited

Loading

[Enhancement] Support profile for only big query #33825

[Enhancement] Support profile for only big query #33825

Conversation

liuyehcf commented Oct 27, 2023 • edited Loading

Background

How

Limitation

Example

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

sonarcloud bot commented Oct 30, 2023

github-actions bot commented Oct 30, 2023

[FE Incremental Coverage Report]

file detail

github-actions bot commented Oct 30, 2023

[BE Incremental Coverage Report]

file detail

github-actions bot commented Oct 30, 2023

github-actions bot commented Oct 30, 2023

mergify bot commented Oct 30, 2023 • edited Loading

✅ Backports have been created

mergify bot commented Oct 30, 2023 • edited Loading

✅ Backports have been created

liuyehcf commented Oct 31, 2023

mergify bot commented Oct 31, 2023 • edited Loading

✅ Backports have been created

liuyehcf commented Oct 31, 2023

mergify bot commented Oct 31, 2023 • edited Loading

✅ Backports have been created

liuyehcf commented Dec 14, 2023 • edited Loading

liuyehcf commented Oct 27, 2023 •

edited

Loading

mergify bot commented Oct 30, 2023 •

edited

Loading

mergify bot commented Oct 30, 2023 •

edited

Loading

mergify bot commented Oct 31, 2023 •

edited

Loading

mergify bot commented Oct 31, 2023 •

edited

Loading

liuyehcf commented Dec 14, 2023 •

edited

Loading