Fix IPCWriter for Sliced BooleanArray #3498
Conversation
Fixes apache#3496.
arrow-ipc/src/writer.rs
Outdated
| // Bools are special because the payload (= 1 bit) is smaller than the physical container elements (= bytes). | ||
| // The array data may not start at the physical boundary of the underlying buffer, so we need to shift bits around. | ||
| assert!(array_data.buffers().len() == 1); | ||
| let buffer = &array_data.buffers()[0]; |
There was a problem hiding this comment.
Could we use Buffer::bit_slice here?
There was a problem hiding this comment.
Ah nice, didn't know that this existed. Fixed.
arrow-ipc/src/writer.rs
Outdated
| } else if matches!(data_type, DataType::Boolean) { | ||
| // Bools are special because the payload (= 1 bit) is smaller than the physical container elements (= bytes). | ||
| // The array data may not start at the physical boundary of the underlying buffer, so we need to shift bits around. | ||
| assert!(array_data.buffers().len() == 1); |
|
Benchmark runs are scheduled for baseline = fb36dd9 and contender = cada9ba. cada9ba is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
Which issue does this PR close?
Closes #3496.
Rationale for this change
It seems that the IPC writer doesn't correctly account for sliced boolean columns. See #3496.
What changes are included in this PR?
New IPC writer routine for bools that shifts data correctly + tests.
Are there any user-facing changes?
IPC for bools no longer broken.