Use SlicesIterator for ArrayData Equality #3198
Conversation
github-actions
bot
added
arrow
viirya
force-pushed
the
use_slices_iterator
branch
from
eca457d to
24788ad
Compare
Tried with
|
arrow-data/src/slices_iterator.rs
Outdated
| /// | ||
| /// Only performant for filters that copy across long contiguous runs | ||
| #[derive(Debug)] | ||
| pub struct SlicesIterator<'a>(BitSliceIterator<'a>); |
There was a problem hiding this comment.
Moving this is a breaking change, as it is exposed publicly - https://docs.rs/arrow-select/latest/arrow_select/filter/struct.SlicesIterator.html
arrow-data/src/slices_iterator.rs
Outdated
|
|
||
| impl<'a> SlicesIterator<'a> { | ||
| pub fn new_from_buffer(values: &'a Buffer, offset: usize, len: usize) -> Self { | ||
| Self(BitSliceIterator::new(values, offset, len)) |
There was a problem hiding this comment.
At this point why not just use BitSliceIterator?
|
|
||
| use super::utils::equal_len; | ||
|
|
||
| pub(crate) const NULL_SLICES_SELECTIVITY_THRESHOLD: f64 = 0.4; |
There was a problem hiding this comment.
How did you come up with 0.4, not saying it is a bad choice, just curious
There was a problem hiding this comment.
It came from benchmarking. equal_nulls_512 has 0.5 null density, and this doesn't improve it. So I pick 0.4 as the threshold.
|
Benchmark runs are scheduled for baseline = 1daf7d3 and contender = f985818. f985818 is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
Which issue does this PR close?
Closes #1600.
Rationale for this change
This improves the performance of long array with less nulls mostly, e.g. the added benchmark:
For array with many nulls, this actually causes regression when using
SlicesIterator, so I only useSlicesIteratorfor the cases with less nulls.What changes are included in this PR?
Are there any user-facing changes?