Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Add function bitmap_from_binary and bitmap_to_binary #35621

Merged
merged 2 commits into from
Nov 24, 2023
Merged

[Feature] Add function bitmap_from_binary and bitmap_to_binary #35621

merged 2 commits into from
Nov 24, 2023

Conversation

trueeyu
Copy link
Contributor

@trueeyu trueeyu commented Nov 23, 2023 •
edited
Loading

Why I'm doing:

Currently, Bitmap import/export needs to be supported. If converted to text format (bitmap_to_string) or base64 (bitmap_to_base64), thre will be some performance overheadand and the output size will be large, so a binary format needs to be defined to facilitate direct writing to parquet files.

What I'm doing:

Implemnt two function:

  • bitmap_from_binary: deserialize binary format to bitmap
  • bitmap_to_binary: serialize bitmap to binary format

The format is consistent with the implementation in segment file.

Later, we will implement a java UDF to facilitate users to parse this format in Hive or application systems.

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 3.2
    • 3.1
    • 3.0
    • 2.5

@trueeyu trueeyu requested a review from a team as a code owner November 23, 2023 02:54
@trueeyu trueeyu changed the title [WIP] [Feature] Add function bitmap_from_binary and bitmap_to_binary [Feature] Add function bitmap_from_binary and bitmap_to_binary Nov 23, 2023
@trueeyu trueeyu force-pushed the bitmap_binary branch 2 times, most recently from 38a5bb3 to 01e3d8b Compare November 23, 2023 03:31
@silverbullet233
Copy link
Contributor

better to add docs of these functions in this pr too.

@trueeyu
Copy link
Contributor Author

trueeyu commented Nov 23, 2023

better to add docs of these functions in this pr too.

I will add the doc in another pr.

LiShuMing
LiShuMing previously approved these changes Nov 23, 2023
View reviewed changes
gensrc/script/functions.py Outdated Show resolved Hide resolved
gensrc/script/functions.py Outdated Show resolved Hide resolved
@trueeyu trueeyu force-pushed the bitmap_binary branch 2 times, most recently from f45f4b0 to a9a55ae Compare November 23, 2023 12:56
imay
imay reviewed Nov 23, 2023
View reviewed changes
gensrc/script/functions.py Outdated Show resolved Hide resolved
be/src/exprs/bitmap_functions.cpp Outdated Show resolved Hide resolved
be/src/exprs/bitmap_functions.cpp Outdated Show resolved Hide resolved
be/src/exprs/bitmap_functions.cpp Outdated Show resolved Hide resolved
test/sql/test_bitmap_functions/T/test_bitmap_functions Show resolved Hide resolved
test/sql/test_bitmap_functions/T/test_bitmap_functions Outdated Show resolved Hide resolved
be/src/exprs/bitmap_functions.cpp Outdated Show resolved Hide resolved
@trueeyu
fix
37e1f7f 
Signed-off-by: trueeyu <[email protected]>
@sonarcloud SonarCloud
Copy link

sonarcloud bot commented Nov 24, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information

warning The version of Java (11.0.21) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more here

@github-actions GitHub Actions
Copy link

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

@github-actions GitHub Actions
Copy link

[BE Incremental Coverage Report]

❌ fail : 1 / 36 (02.78%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 src/exprs/bitmap_functions.cpp 1 36 02.78% [678, 679, 681, 682, 684, 685, 686, 687, 688, 689, 690, 693, 694, 695, 696, 697, 699, 703, 704, 705, 706, 708, 709, 710, 711, 714, 715, 716, 717, 720, 721, 722, 723, 725, 728]

@silverbullet233 silverbullet233 merged commit eb4f691 into StarRocks:main Nov 24, 2023
44 of 45 checks passed
@trueeyu
Copy link
Contributor Author

trueeyu commented Nov 29, 2023

https://github.com/Mergifyio backport branch-3.2

@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 29, 2023 •
edited
Loading

backport branch-3.2

✅ Backports have been created

mergify bot pushed a commit that referenced this pull request Nov 29, 2023
@trueeyu @mergify
[Feature] Add function bitmap_from_binary and bitmap_to_binary (#35621)
36009c0 
Signed-off-by: trueeyu <[email protected]>
(cherry picked from commit eb4f691)
@mergify mergify bot mentioned this pull request Nov 29, 2023
Merged
@trueeyu
Copy link
Contributor Author

trueeyu commented Nov 29, 2023

https://github.com/Mergifyio backport branch-3.1

@trueeyu
Copy link
Contributor Author

trueeyu commented Nov 29, 2023

https://github.com/Mergifyio backport branch-3.0

@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 29, 2023 •
edited
Loading

backport branch-3.1

✅ Backports have been created

@trueeyu
Copy link
Contributor Author

trueeyu commented Nov 29, 2023

https://github.com/Mergifyio backport branch-2.5

@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 29, 2023 •
edited
Loading

backport branch-3.0

✅ Backports have been created

@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 29, 2023 •
edited
Loading

backport branch-2.5

✅ Backports have been created

mergify bot pushed a commit that referenced this pull request Nov 29, 2023
@trueeyu @mergify
[Feature] Add function bitmap_from_binary and bitmap_to_binary (#35621)
c853e39 
Signed-off-by: trueeyu <[email protected]>
(cherry picked from commit eb4f691)
@mergify mergify bot mentioned this pull request Nov 29, 2023
Merged
mergify bot pushed a commit that referenced this pull request Nov 29, 2023
@trueeyu @mergify
[Feature] Add function bitmap_from_binary and bitmap_to_binary (#35621)
119bc5f 
Signed-off-by: trueeyu <[email protected]>
(cherry picked from commit eb4f691)

# Conflicts:
#	be/src/exprs/vectorized/bitmap_functions.cpp
#	be/src/exprs/vectorized/bitmap_functions.h
#	gensrc/script/vectorized/vectorized_functions.py
@mergify mergify bot mentioned this pull request Nov 29, 2023
Closed
mergify bot pushed a commit that referenced this pull request Nov 29, 2023
@trueeyu @mergify
[Feature] Add function bitmap_from_binary and bitmap_to_binary (#35621)
2cde96c 
Signed-off-by: trueeyu <[email protected]>
(cherry picked from commit eb4f691)

# Conflicts:
#	be/src/exprs/bitmap_functions.cpp
#	be/src/exprs/bitmap_functions.h
#	gensrc/script/functions.py
@mergify mergify bot mentioned this pull request Nov 29, 2023
Closed
trueeyu added a commit to trueeyu/starrocks that referenced this pull request Nov 29, 2023
@trueeyu
[Feature] Add function bitmap_from_binary and bitmap_to_binary (StarR…
338de62 
…ocks#35621)

Signed-off-by: trueeyu <[email protected]>
@trueeyu trueeyu mentioned this pull request Nov 29, 2023
Merged
wanpengfei-git pushed a commit that referenced this pull request Nov 30, 2023
@trueeyu @wanpengfei-git
[Feature] Add function bitmap_from_binary and bitmap_to_binary (#35621)
05ac129 
Signed-off-by: trueeyu <[email protected]>
(cherry picked from commit eb4f691)
wanpengfei-git pushed a commit that referenced this pull request Nov 30, 2023
@trueeyu @wanpengfei-git
[Feature] Add function bitmap_from_binary and bitmap_to_binary (#35621)
628b9fe 
Signed-off-by: trueeyu <[email protected]>
(cherry picked from commit eb4f691)
kangkaisen pushed a commit that referenced this pull request Nov 30, 2023
@trueeyu
[Feature] Add function bitmap_from_binary and bitmap_to_binary (#35621)…
89b5fe7 
… (#36092)

Signed-off-by: trueeyu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@starrocks-cr starrocks-cr[bot] starrocks-cr[bot] left review comments

@silverbullet233 silverbullet233 silverbullet233 approved these changes

@imay imay imay approved these changes

@LiShuMing LiShuMing LiShuMing left review comments

Assignees

@trueeyu trueeyu

Labels
backport_ok
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@trueeyu @silverbullet233 @imay @LiShuMing @wanpengfei-git