pl.read_ipc(..., columns=specific_column_order, use_pyarrow=False) does not preserve the column order .

[ghuls]

Are you using Python or Rust?

Python.

What version of polars are you using?

git commit: e8f6b0f

What operating system are you using polars on?

CentOS 7

Describe your bug.

pl.read_ipc(..., columns=specific_column_order, use_pyarrow=False) does not preserve the column order specified by the user (as it is sorted by column indices when passing to arrow2 read IPC function).

What are the steps to reproduce the behavior?

In [28: import polars as pl
]
In [29]: df = pl.DataFrame([
    ...:     pl.Series("a", [2, 5]),
    ...:     pl.Series("b", [4, 3]),
    ...:     pl.Series("c", [1, 6]),
    ...:     pl.Series("d", [7, 9]),
    ...: ])

In [30]: df.to_ipc('test.feather')

In [31]: df_read = pl.read_ipc('test.feather', columns=["a", "b", "c", "d"], use_pyarrow=False)

In [32]: df_read_out_of_order = pl.read_ipc('test.feather', columns=["c", "b", "d", "a"], use_pyarrow=False)

In [33]: df
Out[33]: 
shape: (2, 4)
┌─────┬─────┬─────┬─────┐
│ a   ┆ b   ┆ c   ┆ d   │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╪═════╡
│ 2   ┆ 4   ┆ 1   ┆ 7   │
├╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌┤
│ 5   ┆ 3   ┆ 6   ┆ 9   │
└─────┴─────┴─────┴─────┘

In [34]: df_read
Out[34]: 
shape: (2, 4)
┌─────┬─────┬─────┬─────┐
│ a   ┆ b   ┆ c   ┆ d   │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╪═════╡
│ 2   ┆ 4   ┆ 1   ┆ 7   │
├╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌┤
│ 5   ┆ 3   ┆ 6   ┆ 9   │
└─────┴─────┴─────┴─────┘

In [35]: df_read_out_of_order
Out[35]: 
shape: (2, 4)
┌─────┬─────┬─────┬─────┐
│ a   ┆ b   ┆ c   ┆ d   │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╪═════╡
│ 2   ┆ 4   ┆ 1   ┆ 7   │
├╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌┤
│ 5   ┆ 3   ┆ 6   ┆ 9   │
└─────┴─────┴─────┴─────┘

In [36]: df_read_out_of_order_pyarrow = pl.read_ipc('test.feather', columns=["c", "b", "d", "a"], use_pyarrow=True)

In [37]: df_read_out_of_order_pyarrow
Out[37]: 
shape: (2, 4)
┌─────┬─────┬─────┬─────┐
│ c   ┆ b   ┆ d   ┆ a   │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╪═════╡
│ 1   ┆ 4   ┆ 7   ┆ 2   │
├╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌┤
│ 6   ┆ 3   ┆ 9   ┆ 5   │
└─────┴─────┴─────┴─────┘

[jorgecarleitao]

Maybe we should address this in arrow2; atm it does not support projections over IPC with an order different from the one on the file, but I can't find a reason why it shouldn't; it is just a column swap before creating the record batch.

[ritchie46]

Maybe we should address this in arrow2; atm it does not support projections over IPC with an order different from the one on the file, but I can't find a reason why it shouldn't; it is just a column swap before creating the record batch.

Then we ditch the sort 👍

[ghuls]

@jorgecarleitao I just retested it with the latest polars version and it seems it is still not supported in the latest arrow2.

[jorgecarleitao]

Being addressed upstream: jorgecarleitao/arrow2#961