[Enhancement] Optimize the inner-join performance of nest-loop-join when the right table is relatively small

[trueeyu]

Why I'm doing:

For nest-loop-join for inner join, if the right table is small such as one row,
In order to generate intermediate chunks, only a small number of rows will be appended at a time, which is very time-consuming.

What I'm doing:

For inner-join, we will generate the intermediate chunk base the larger chunk of build or probe.

CREATE TABLE `lineorder` (
  `lo_orderkey` int(11) NOT NULL COMMENT "",
  `lo_linenumber` int(11) NOT NULL COMMENT "",
  `lo_custkey` int(11) NOT NULL COMMENT "",
  `lo_partkey` int(11) NOT NULL COMMENT "",
  `lo_suppkey` int(11) NOT NULL COMMENT "",
  `lo_orderdate` int(11) NOT NULL COMMENT "",
  `lo_orderpriority` varchar(16) NOT NULL COMMENT "",
  `lo_shippriority` int(11) NOT NULL COMMENT "",
  `lo_quantity` int(11) NOT NULL COMMENT "",
  `lo_extendedprice` int(11) NOT NULL COMMENT "",
  `lo_ordtotalprice` int(11) NOT NULL COMMENT "",
  `lo_discount` int(11) NOT NULL COMMENT "",
  `lo_revenue` int(11) NOT NULL COMMENT "",
  `lo_supplycost` int(11) NOT NULL COMMENT "",
  `lo_tax` int(11) NOT NULL COMMENT "",
  `lo_commitdate` int(11) NOT NULL COMMENT "",
  `lo_shipmode` varchar(11) NOT NULL COMMENT ""
) ENGINE=OLAP 
DUPLICATE KEY(`lo_orderkey`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`lo_orderkey`) BUCKETS 192 
PROPERTIES (
"replication_num" = "1",
"colocate_with" = "groupa1",
"in_memory" = "false",
"enable_persistent_index" = "false",
"replicated_storage" = "true",
"compression" = "LZ4"
);

select count(*) from lineorder;
+-----------+
| count(*)  |
+-----------+
| 143999468 |
+-----------+

CREATE TABLE `t1` (
  `c1` int(11) NULL COMMENT "",
  `c2` int(11) NULL COMMENT ""
) ENGINE=OLAP 
DUPLICATE KEY(`c1`)
DISTRIBUTED BY HASH(`c1`) BUCKETS 1 
PROPERTIES (
"replication_num" = "1",
"in_memory" = "false",
"enable_persistent_index" = "false",
"replicated_storage" = "true",
"fast_schema_evolution" = "true",
"compression" = "LZ4"
); 

select * from t1;
+------+------+
| c1   | c2   |
+------+------+
|    1 |    1 |
+------+------+

CREATE TABLE `t2` (
  `c1` int(11) NULL COMMENT "",
  `c2` bitmap BITMAP_UNION NULL COMMENT ""
) ENGINE=OLAP 
AGGREGATE KEY(`c1`)
DISTRIBUTED BY HASH(`c1`) BUCKETS 1 
PROPERTIES (
"replication_num" = "1",
"in_memory" = "false",
"enable_persistent_index" = "false",
"replicated_storage" = "true",
"fast_schema_evolution" = "true",
"compression" = "LZ4"
); 

select c1, bitmap_count(c2) from t2;
+------+------------------+
| c1   | bitmap_count(c2) |
+------+------------------+
|    1 |         30000000 |
+------+------------------+

Case 1:

select count(*) from lineorder join [broadcast] t1 on lo_partkey>c2;

Time: 11.974s -> 0.54s
Mem: 5.363M -> 4M

Case 2:

select count(*) from lineorder join [broadcast] t2 on bitmap_contains(c2, lo_orderkey);

Time: 20.680s -> 9.428s
Mem: 30M -> 30M

Fixes #issue

What type of PR is this:

• BugFix
• Feature
• Enhancement
• Refactor
• UT
• Doc
• Tool

Does this PR entail a change in behavior?

• Yes, this PR will result in a change in behavior.
• No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

• Interface/UI changes: syntax, type conversion, expression evaluation, display information
• Parameter changes: default values, similar parameters but with different default values
• Policy changes: use new policy to replace old one, functionality automatically enabled
• Feature removed
• Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

• I have added test cases for my bug fix or my new feature
• This pr needs user documentation (for new or modified features or behaviors)
  • I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

• I have checked the version labels which the pr will be auto-backported to the target branch
  • 3.2
  • 3.1
  • 3.0
  • 2.5

[github-actions]

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

[github-actions]

[BE Incremental Coverage Report]

❌ fail : 0 / 62 (00.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	src/exec/pipeline/nljoin/nljoin_probe_operator.cpp	0	62	00.00%	[189, 194, 196, 198, 200, 201, 203, 204, 205, 208, 209, 210, 216, 218, 220, 222, 223, 225, 226, 229, 233, 413, 414, 416, 417, 418, 419, 420, 422, 425, 426, 427, 429, 430, 432, 434, 437, 438, 439, 440, 441, 442, 444, 445, 446, 447, 448, 452, 453, 454, 455, 456, 457, 459, 460, 461, 462, 463, 471, 497, 633, 662]

[trueeyu]

https://github.com/Mergifyio backport branch-3.2

[mergify]

backport branch-3.2

✅ Backports have been created

• #35383 [Enhancement] Optimize the inner-join performance of nest-loop-join when the right table is relatively small (backport #35003) has been created for branch branch-3.2

[trueeyu]

https://github.com/Mergifyio backport branch-3.1

[mergify]

backport branch-3.1

✅ Backports have been created

• #35382 [Enhancement] Optimize the inner-join performance of nest-loop-join when the right table is relatively small (backport #35003) has been created for branch branch-3.1

[trueeyu]

https://github.com/Mergifyio backport branch-3.0

[trueeyu]

https://github.com/Mergifyio backport branch-2.5

[mergify]

backport branch-3.0

✅ Backports have been created

• #35384 [Enhancement] Optimize the inner-join performance of nest-loop-join when the right table is relatively small (backport #35003) has been created for branch branch-3.0

[mergify]

backport branch-2.5

✅ Backports have been created

• #35385 [Enhancement] Optimize the inner-join performance of nest-loop-join when the right table is relatively small (backport #35003) has been created for branch branch-2.5 but encountered conflicts