-
Notifications
You must be signed in to change notification settings - Fork 762
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
remove buffer in block_reader of fuse store #4230
Conversation
This pull request is being automatically deployed with Vercel (learn more). 🔍 Inspect: https://vercel.com/databend/databend/81YwDsT8wUDK4aaHFox4Q5TFbNNJ [Deployment for 7373fc5 canceled] |
Thanks for the contribution! Please review the labels and make any necessary changes. |
Hi, as #4203 is going to merge, how about wait a bit so that we don't need to resolve the conflict? |
@Xuanwo ok, this is a small pr, I will wait for you to merge first |
Thanks for the patients! |
OpenDAL introduces an API redesign. Welcome any feedback for that! |
Merged! |
@@ -63,7 +63,6 @@ impl FuseTable { | |||
|
|||
let part_stream = futures::stream::iter(iter); | |||
|
|||
let read_buffer_size = ctx.get_settings().get_storage_read_buffer_size()?; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does ctx.get_storage_read_buffer_size not used anymore?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
still in use
@BohuTANG not needed for the current parquet2 implementation, but this is function is also used to read our JSON metafile. need a little refactor. |
I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/
Summary
do not need buffer since "we execute exactly 1 seek and 1 read on them."
https://github.com/jorgecarleitao/arrow2/blob/3d528c99589e96f0539de4c07b11843fa22f23ac/examples/parquet_read_async.rs#L31
a buffer with column chunk length is allocated each time.
https://github.com/jorgecarleitao/arrow2/blob/3d528c99589e96f0539de4c07b11843fa22f23ac/src/io/parquet/read/row_group.rs#L126
thanks to @dantengsky
#4212 (comment)
Changelog
Related Issues
Test Plan
Unit Tests
Stateless Tests