fix: cast support fields nested in lists and maps #2541
Conversation
|
ACTION NEEDED delta-rs follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
HawaiianSpork
changed the title
|
Please add also a bunch of python tests |
@ion-elgreco I'm confused as to what a Python set of tests would do that the tests in Rust don't already do? 😕 |
We mostly need to check the reader side with pyarrow datasets |
|
@HawaiianSpork thanks for the PR! can you rebase please so we can merge |
The current implementation of cast only works structs nested in structs. This PR adds supports for structs nested in other types. This PR also prevents cast from adding nullable column if the field is not nullable.
HawaiianSpork
force-pushed
the
cast-with-more-containers
branch
from
b0fe32c to
457f967
Compare
|
@ion-elgreco thank you. Sorry, I did not get around to writing python tests. The code has been rebased. |
Description
The current implementation of cast only works for structs nested in structs. This PR adds supports for structs contained in other types (lists and maps). This PR also prevents cast from adding nullable column if the field is not nullable, instead it will throw an error.
Note: This is only a partial solution which would let you merge schema with nested missing columns, it does not allow delta-rs to read the merged schema (though Spark can). To read the merged schema will require another change where delta-rs defines its own datafusion parquet schemaAdapter.