Writing Parquet file with a Dictionary array with Compression enabled create invalid files

[Stilgar84]

Hi,

I am using arrows2 to create a simple record batch which contains one Dictionary array. when writing to parquet with Snappy or gzip, the output cannot be read back with pyarrows. If I remove either the compression or the dictionary, there is no problem.

pyarrows errors:

• Corrupt snappy compressed data
• GZipCodec failed: incorrect header check

[jorgecarleitao]

Hey! Thanks for your interest. :) I think it is solved by jorgecarleitao/arrow2#523 . Could you try the latest main from arrow2?

[Stilgar84]

Thanks, that fixed it. I checked the opened issue, but not the closed ones :)

[jorgecarleitao]

btw, I keep track of performance benchmarks here and here. If you find something that does not fit the general numbers, please let us know: always looking for edge cases and the like ^_^