[Question] Why DataFusion is shipped with arrow version 9.1.0 on crates.io ?

[Cheappie]

I wonder why DataFusion is shipped with arrow version 9.1.0 on crates.io, could anybody tell me how it ends up that way even though in every Cargo.toml within DataFusion repository newest Arrow version is used, precisely 13.0.0 ?

[andygrove]

The version of DataFusion on crates.io was released several months ago with a dependency on arrow-rs 9.0.0 because that was the latest version at that time. We are planning on releasing DataFusion 8.0.0 in the next week or two and that will use arrow-rs 13.0.0 (or whatever the latest version is at the time).

[andygrove]

We are also planning on having more frequent releases going forward, as discussed in #2327

[Cheappie]

The version of DataFusion on crates.io was released several months ago with a dependency on arrow-rs 9.0.0 because that was the latest version at that time. We are planning on releasing DataFusion 8.0.0 in the next week or two and that will use arrow-rs 13.0.0 (or whatever the latest version is at the time).

Thank you Andy for clearing this out. In such case I will try to use master directly.

I saw on crates.io that the latest version was published 19 days ago, thats why I couldn't wrap my head around why it uses so old version of Arrow. I suppose that it still lags behind master quite far.

I am not sure if that change apache/arrow-rs#1667 will be accepted, but If It will pass then I would hope that It will be a part of next DataFusion release.

[tustvold]

FWIW IOx tracks a git version pin for DataFusion, in order to get upstream changes quicker - https://github.com/influxdata/influxdb_iox/blob/main/datafusion/Cargo.toml

This of course means we are exposed to more churn though, so your mileage may vary

[alamb]

Next release is imminent; #2392

I think this ticket has been answered so closing it down