Extend in place! 💯

[ritchie46]

This PR is quite a big deal. Until now polars/arrow memory was completely immutable. If we did and append, we simply added an array chunk to the list of chunks (sort of a linked list). This yielded very fast appends, but is detrimental for query performance, because the chunks add a lot of indirection.

Especially use cases where you have rows coming in on a very slow pace and you want to do querys between the updates. For instance in online learning cases, Polars was not the right tool for the job, as you would need to call a rechunk to get optimal peformance which is a complete reallocation of your table! Very expensive.

With this change, we can now extend the DataFrame/Series and write to the same memory allocations. This might still reallocate, but given exponential growth strategies, this operation is amortized O(1).

There is of course no magic. We can only write to the same memory iff

• we are the only owner (The series are not shared with another dataframe)
• the memory is not allocated by pyarrow

[ritchie46]

@jorgecarleitao @houqp FYI

[houqp]

very cool!

[jorgecarleitao]

Brutal. Always innovating!

fyi @wesm @pitrou @kou @andygrove. This uses copy on write - it checks at runtime whether we are the only owners of the array and, if yes, we take exclusive mutable ownership of the buffer / array.

This was proposed by @sundy-li here jorgecarleitao/arrow2#741, inspired by what clickhouse is doing and implemented by @ritchie46 here: jorgecarleitao/arrow2#794