Error while writing Pandas DataFrame to Delta Lake (S3)

[vinamrgrover]

write_deltalake isn't working as expected

I encountered an error while calling write_deltalake method:

ValueError : you must provide schema if data is iterable

Even though it worked perfectly with a PyArrow Table, it didn't work for Pandas DataFrame.

How to reproduce it:

# To reproduce it, just try to write a Pandas DataFrame to an S3 Bucket

write_deltalake(
    <your_s3_url>,
    data = df,
    storage_options = storage_options,
    overwrite_schema = True,
    mode = 'overwrite'
)

[ion-elgreco]

@vinamrgrover please provide the deltalake version you're using, also share a minimal reproducible example

[vinamrgrover]

Version : 0.15.0

I already shared a reproducible example above

[ion-elgreco]

@vinamrgrover a reproducible example should also include a sample dataframe..

[ion-elgreco]

You are likely passing something that is not a pandas dataframe, I can write a pandas dataframe with write_deltalake.

import pandas as pd
from deltalake import write_deltalake
df = pd.DataFrame({'foo':['test']})
write_deltalake('test_table_PATH', data=df, mode='overwrite', overwrite_schema=True)

[vinamrgrover]

Please don't mark it as completed, I better know I have passed Pandas DataFrame in write_deltalake method. I think the issue is with the _has_pandas variable within writer.py module. Try executing this in a new virtual environment