Parquet: Snappy buffer size limitation.

[ritchie46]

Writing large parquet files can lead to:

ArrowError(ExternalFormat("External format error: underlying snap error: snappy: input buffer (size = 68397383472) is larger than allowed (size = **4294967295)**

I believe we already discussed this earlier in a polars issue, but I cannot reproduce it. I open this issue to explore ways to circumvent it.
I assume we can prevent this by passing smaller Chunks to the writer?

Is there a way upfront to estimate the size of a buffer that will be send to snappy?

[jorgecarleitao]

We need to support splitting the arrays in multiple parquet pages when writing, since each page is compressed independently

[jorgecarleitao]

Alternatively, split the chunk in smaller parts ^^

[ritchie46]

Alternatively, split the chunk in smaller parts ^^

I have been thinking about this solution. Is what we send to snappy the whole row group, or separate columns?

I am thinking of a proxy to determine the needed chunk size.

Something like this? estimated_size / (snappy_limit * n_columns)