invalid written parquet file of nested structures. (Mixing list with structs)

[ritchie46]

Arrow2 can read the written file, but arrow fails with:

OSError: Malformed levels. min: 0 max: 3 out of range.  Max Level: 2

MWE

import polars as pl
import pyarrow.parquet as pq

pl.from_records(
    [
        dict(
            id=1,
            list_of_structs_col=[
                dict(a=10, b=[10, 11, 12]),
                dict(a=11, b=[13, 14, 15]),
            ],
        ),
        dict(
            id=2,
            list_of_structs_col=[
                dict(a=44, b=[12]),
            ],
        ),
    ]
).write_parquet("/tmp/out.parquet")

print(pl.read_parquet("/tmp/out.parquet"))  # succeeds
print(df = pq.read_table("/tmp/out.parquet"))  # fails

[ritchie46]

Also noted that though arrow2 can read the file, the content that is read is incorrect.

[ritchie46]

The issue is the in the repetition levels.

They are [0, 3, 3, 1, 3, 3, 0, ]. And should be [0, 2, 2, 1, 2, 2, 0, ].

Still have to figure out how the conversion works to be able to fix it.