Lance File Format 2.2

[westonpace]

As 2.1 is wrapping up I am using this issue to list features which will not make the cut for 2.1. They are candidate features for a 2.2 release.

Run length encoding

Run length encoding can be supported in both the mini-block and full-zip encodings. In the mini-block encoding scheme we don't have to do anything special. In the full-zip scheme we would need to store the run offsets into the repetition index.

FSL<LIST<..>> and LIST<FSL<..>>

These data types are uncommon but possible. For completeness sake we should support them.

Variant data type

Variant data types are becoming popular and parquet is adding basic support. It would be good to understand what the use cases are for variant data types and add support in Lance too.j

Union data type

Another arrow type that we don't support. Unclear if this would be more useful than variant type.