You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The initial version should only support reading the entire file, then future work can be done on the actual streaming of data if there is enough demand.
I'm keen to get this into polars, as Snowflake uses this for their response format and would be awesome to get it in for reading data straight from SF into Polars (with minimal massaging of data).
I've started looking into this, and the major blocker I can see is projections.
This is crossposted from pola-rs/nodejs-polars#109 , to track the feature of reading and writing Arrow streaming files.
For context about Arrow Streams, please read https://jorgecarleitao.github.io/arrow2/io/ipc_stream_read.html .
Here is a quick primer about the streaming files from Arrow: https://arrow.apache.org/docs/python/ipc.html
The initial version should only support reading the entire file, then future work can be done on the actual streaming of data if there is enough demand.
I'm keen to get this into polars, as Snowflake uses this for their response format and would be awesome to get it in for reading data straight from SF into Polars (with minimal massaging of data).
I've started looking into this, and the major blocker I can see is projections.
In arrow2, projections are not supported here: https://github.com/jorgecarleitao/arrow2/blob/main/src/io/ipc/read/stream.rs#L185
So we will need to build the projection from the chunks.
Thoughts?
The text was updated successfully, but these errors were encountered: