Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Arrow: Parquet does not support writing empty structs when creating checkpoint #2622

Closed
echai58 opened this issue Jun 24, 2024 · 0 comments · Fixed by #2627
Closed

Arrow: Parquet does not support writing empty structs when creating checkpoint #2622

echai58 opened this issue Jun 24, 2024 · 0 comments · Fixed by #2627
Labels
bug Something isn't working

Comments

@echai58
Copy link

echai58 commented Jun 24, 2024

Environment

Delta-rs version: 0.18.1

Binding: python


Bug

What happened:
When trying to create a checkpoint on a table with one non-partition column of binary type, I get the error:

OSError: Arrow: Parquet does not support writing empty structs

I found this PR #2125, whose intention seemed to be to fix exactly this, but doesn't seem to be the case.

What you expected to happen:
To be able to create the checkpoint.

How to reproduce it:

from deltalake import DeltaTable, write_deltalake
import pandas as pd

write_deltalake(
    "test",
    pd.DataFrame.from_dict(
        {
            "p": [1],
            "k": [b'a'],
        }
    ),
    partition_by=["p"],
)

DeltaTable("test").create_checkpoint()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant