This repository has been archived by the owner on Feb 18, 2024. It is now read-only.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a follow-up to https://lists.apache.org/thread/3wh51ml8zt00131fgmxj1lqg92dlgj41 and related to #1197
The gist is that, in my opinion, to read mmapped Arrow IPC files soundly, the buffers need to be written with a bit alignment consistent with their types. Currently the spec does not to support this. As such,
i128
andi256
can't be soundly read viammap
with 64bit alignment, and no type other thani8
andu8
can be soundly read in 8bit aligned files.We currently write files whose every buffer is 8bit aligned, which restricts zero-copy reading every type other than
i8
andu8
. This PR changes the write alignment to 64 bits. Note that this has virtually no impact to performance or file size, but is a requirement to mmap IPC files as per #1197 (and in general).