Written flatbuffer can't be read by other flatbuffer readers

[jorgecarleitao]

I think that there is a bug somewhere in the writer.

• planus can read C++- and Rust- generated arrow files (+30 different files)
• planus can read planus-generated arrow files (i.e. write data to flatbuffers -> read data from flatbuffers -> check data is equal) (+30 different files)
• C++ and flatbuffers crate implementations can't read flatbuffers written by planus

Repro steps (sorry, not very minimal atm):

git clone [email protected]:jorgecarleitao/arrow2.git
cd arrow2

# write using flatbuffers crate:
git checkout main
cargo run --example ipc_file_write --features io_ipc baseline.arrow

# read flatbuffers-generated using flatbuffers crate works as intended
cargo run --example ipc_file_read --features io_ipc,io_print baseline.arrow

git checkout planus
# read flatbuffers-generated using planus works as intended
cargo run --example ipc_file_read --features io_ipc,io_print baseline.arrow

# write planus-generated with planus:
cargo run --example ipc_file_write --features io_ipc planus.arrow

# read planus-generated with planus works as intended
cargo run --example ipc_file_read --features io_ipc,io_print planus.arrow

# read planus-generated with flatbuffers fails
git checkout main
cargo run --example ipc_file_read --features io_ipc,io_print planus.arrow

# read flatbuffers-generated with C++ flatbuffers works as intended:
python3 -m venv venv
venv/bin/pip install -U pip && venv/bin/pip install pyarrow
venv/bin/python -c "import pyarrow.ipc; batch = pyarrow.ipc.open_file('baseline.arrow').get_batch(0)"

# read planus-generated with C++ flatbuffers fails:
python3 -m venv venv
venv/bin/pip install -U pip && venv/bin/pip install pyarrow
venv/bin/python -c "import pyarrow.ipc; batch = pyarrow.ipc.open_file('planus.arrow').get_batch(0)"

Error with Rust's flatbuffers:

Error: OutOfSpec("Unable to get root as footer: MissingNullTerminator { range: 224..225, error_trace: ErrorTrace([TableField { field_name: \"name\", position: 164 }, VectorElement { index: 0, position: 96 }, TableField { field_name: \"fields\", position: 80 }, TableField { field_name: \"schema\", position: 12 }]) }")

Error with C++ implementation: Verification of flatbuffer-encoded Footer failed..

The fact that planus can read both the flatbuffers from Rusts flatbuffers and C++ implementation and the planus-generated flatbuffers hints that it is the writer, not the reader, that has an issue here - it is writing something that planus can read but others can't.

[TethysSvensson]

This looks like an instance of #16. I will try to add a fix for it tomorrow.

[TethysSvensson]

@jorgecarleitao I've merged #36, so if you could take another look that would be great.

Do you need me to do a new release or are you okay with depending directly on the main branch on git?

[jorgecarleitao]

That was freaking fast for a +2k LOC PR xD

Just did a quick test with the latest main and the issue has been fixed 🎉

I am running the full suite against C++, Go, JS and C# and will report back if anything else pops up.

[jorgecarleitao]

All tests passed 💯💯💯 🚀🚀🚀🚀🚀 - jorgecarleitao/arrow2#732