Added and_scalar and or_scalar for boolean

[silathdiir]

Close #663

[codecov]

Codecov Report

Merging #707 (ad36ece) into main (f64339c) will increase coverage by 0.41%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #707      +/-   ##
==========================================
+ Coverage   69.77%   70.19%   +0.41%     
==========================================
  Files         303      309       +6     
  Lines       16855    16815      -40     
==========================================
+ Hits        11761    11803      +42     
+ Misses       5094     5012      -82     

Impacted Files	Coverage Δ
src/compute/boolean.rs	95.34% <100.00%> (+2.01%)	⬆️
src/io/parquet/read/primitive/utils.rs	72.72% <0.00%> (-18.19%)	⬇️
src/io/avro/read/header.rs	84.61% <0.00%> (-15.39%)	⬇️
src/io/avro/read/block.rs	59.45% <0.00%> (-11.97%)	⬇️
src/datatypes/schema.rs	32.00% <0.00%> (-5.94%)	⬇️
src/io/parquet/read/mod.rs	38.84% <0.00%> (-5.92%)	⬇️
src/io/json/read/deserialize.rs	68.42% <0.00%> (-5.46%)	⬇️
src/io/avro/read/deserialize.rs	67.22% <0.00%> (-5.43%)	⬇️
src/io/avro/read_async/block.rs	78.12% <0.00%> (-4.64%)	⬇️
src/io/avro/read/mod.rs	80.00% <0.00%> (-1.25%)	⬇️
... and 88 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f64339c...ad36ece. Read the comment docs.

[jorgecarleitao]

Thank you so much for this! The amount of testing that is here is really impressive.

I think that there is an optimization here:

• When the scalar is true, A & true = A and A | true = A.
• When the scalar is false, A & false = false and A | false = A

i.e. we do not need to compute this item by item; we just need to clone and/or initialize all items as false / unset accordingly. This should be orders of magnitude faster since it is not O(N).

What do you think?

[silathdiir]

I think that there is an optimization here:

• When the scalar is true, A & true = A and A | true = A.
• When the scalar is false, A & false = false and A | false = A

i.e. we do not need to compute this item by item; we just need to clone and/or initialize all items as false / unset accordingly. This should be orders of magnitude faster since it is not O(N).

What do you think?

Yes. That should be great. I will update the code. Thanks.

[jorgecarleitao]

Great one. Left two minor suggestions :)

[jorgecarleitao]

Suggested change

	let values = Bitmap::from_trusted_len_iter(std::iter::repeat(true).take(array.len()));
	let values = Bitmap::from_len_zeroed(array.len());

[silathdiir]

For this or version, I want to initialize a Bitmap with all true bits to implement A | true = true. Is that right? Thanks.

[jorgecarleitao]

Oh, yes, you are right!

Unfortunately we do not have a specific method for that, but it is worth checking MutableBitmap::extend_constant here, since it sets 8 bits per write, still much faster than from_trusted_len_iter ^_^

[jorgecarleitao]

For context, the reason we have all these seemly similar APIs is that for a single API (like rust's Vec::extend) we need trait specialization in the stable channel. This is why we currently name every method differently.

[silathdiir]

Thanks for instruction. I updated to use MutableBitmap:: extend_constant. Thanks.

[jorgecarleitao]

Merged. Thanks a lot, @silathdiir!