Panic on compression when writing large parquet files.

[SimonSchneider]

What language are you using?

Rust

Which feature gates did you use?

latest:
polars = { git = "https://github.com/pola-rs/polars", rev = "91e089b920b0928eb1dbfa0c1bd7ba0935e92f8f", features = ["lazy", "is_in", "parquet", "json", "strings", "sort_multiple", "round_series"] }

Describe your bug.

Writing "large" parquet files fails

What are the steps to reproduce the behavior?

    #[test]
    fn write_large_parquet() -> Result<()> {
        let n = 1000;
        let vec: Vec<i32> = (0..n).collect();
        let mut df = DataFrame::new(vec![Series::new("num", vec)])?;
        let mut buf = Cursor::new(Vec::new());
        ParquetWriter::new(&mut buf).finish(&mut df)?;
        Ok(())
    }

What is the actual behavior?

Running with n = 100 the test passes but for n=1000 it panics with:

running 1 test
Error: External format error: underlying IO error: Compression failed

I believe this to be fixed in jorgecarleitao/parquet2#140 but Arrow and Polars would have to be updated but I'm not sure if this is actually the cause.

Feel free to close this issue if you don't think it gives value to track this on the Polars repo.

[ritchie46]

Yes, this should already been fixed. @jorgecarleitao issued a new parquet2 release a few days ago. It should be automatically used if you remove your lockfile.

[SimonSchneider]

I'll give it a try, must have missed that, wasn't aware that it was possible to have to ditch the lock file

[SimonSchneider]

seems it was indeed the lockfile that had to be removed, sorry about the false positive. Was not aware that the lockfile could cause issues like that.

[ritchie46]

Was not aware that the lockfile could cause issues like that.

No worries. A lockfile fixes the versions and does not automatically take latest patch releases. I believe cargo update can also help.