Pyarrow engine not supporting schema overwrite with Append mode

[gprashmi]

Environment

Delta-rs version: 0.17.4

Bug

We have a pandas df with 66 columns that is being written to a delta table with a pre-defined schema in pyarrow engine and now we have a new df with 67 columns, and the schema_mode = overwrite is not supported with 'append' mode.

Below is an basic example df of updating table schema with pyarrow engine and append mode: and the error we see while updating table schema

# Define the new schema with additional columns
new_columns = [pa.field('new_column', pa.float64())]

# Read the existing Delta table
delta_table = DeltaTable(delta_table_path)

# Get the schema of the existing table
existing_schema = delta_table.schema().to_pyarrow()
# Combine the existing schema fields with the new columns
updated_schema = pa.schema(list(existing_schema) + new_columns)

# Create a new DataFrame with the updated schema and add data
updated_data = {
    'column1': ['d', 'e', 'f'],
    'column2': [4, 5, 6],
    'dayhour': ['2024-07-01 03:00', '2024-07-01 04:00', '2024-07-01 05:00'],
    'new_column': [0.4, 0.5, 0.6]
}
updated_df = pd.DataFrame(updated_data)

# Convert the updated DataFrame to a PyArrow Table
updated_table = pa.Table.from_pandas(updated_df, schema=updated_schema)

# Write the initial Delta table
write_deltalake(
    table_or_uri=delta_table_path,
    data=updated_table.to_pandas(),
    schema=updated_schema,
    partition_by=[dayhour_partition_column],
    schema_mode='merge',
    mode="append",
    storage_options={"AWS_S3_ALLOW_UNSAFE_RENAME": "true"},
)

It would be great if we can have overwrite the existing schema of the delta table with append mode that does not effect the existing data. Can you please let me know if we can update schema while appending data?

[rtyler]

👋 The schema_mode='merge' parameter is not, and will likely never be supported with the pyarrow engine. Is there a reason why the engine='rust' cannot be used? That's the directly we're trying to move towards.

Additionally I wanted to mention that:

    storage_options={"AWS_S3_ALLOW_UNSAFE_RENAME": "true"},

this options introduces table corruption risk if there are ever two processes which try to concurrently modify the same Delta table.

[gprashmi]

The reason for not using engine='rust' is we want to have this delta table mapped to Trino (for Grafana integration). But the mapping between delta table with rust and Trino was not compatible and throwing the below error. So we had to move to Pyarrow with which we could map to Trino.

[gprashmi]

@g12-al

[gprashmi]

Also, you mentioned about storage_options introducing table corruption risk if there are ever two processes which try to concurrently modify the same Delta table
storage_options={"AWS_S3_ALLOW_UNSAFE_RENAME": "true"},

so we initially had optimize.compact() on the delta table and this was causing the partition URL encoding to have spaces randomly (opened a github issue for this: #2634), could the 'storage_options' be associated to this random URL encoding with spaces?

[gprashmi]

@rtyler Any update on the above comment for using the pyarrow engine for writing data.

[ion-elgreco]

The reason for not using engine='rust' is we want to have this delta table mapped to Trino (for Grafana integration). But the mapping between delta table with rust and Trino was not compatible and throwing the below error. So we had to move to Pyarrow with which we could map to Trino.

We are going to deprecate the pyarrow engine eventually, so schema evolution won't be supported there.

If you are encountering issues with Trino when reading tables created by rust engine, then please create a separate issue with an MRE