Deleting large number of records fails with no error message

[jpambrun-vida]

Environment

Delta-rs version: delta-rs.0.19.0

Binding: python

Environment:

• Cloud provider: na
• OS: linux
• Other:

---

Bug

What happened:

Calling delete() fails with a long query without any message. The exit code is 1.

What you expected to happen:

Given that we need to delete in a single command to void creating many table version I expect delete() to take an arbitrary long query.

How to reproduce it:

In this example, I create a table of 100,000 rows. Think if x and and ID and I then wanted to delete 10% by id.
It seems to fail when trying to delete ~1600 items.

from deltalake import DeltaTable, write_deltalake
import pandas as pd
import shutil

print(f"deleting table")
shutil.rmtree("./deltatable")


print(f"creating table")
df = pd.DataFrame({'x': [f"{i}" for i in range(100000)]})
write_deltalake('./deltatable', df)


print(f"deleting rows")
dt = DeltaTable("./deltatable")
query = ' or '.join([f"x = '{i}'" for i in range(16000) if i % 10 == 0]) # fails at 16000, works at 15000
del_metrics = dt.delete(query)
print(del_metrics)


_dt = DeltaTable("./deltatable")
print(_dt.history())

More details:

[ion-elgreco]

You are running OOM, at least when ran it that's what happened.

If you rewrite your sql delete statement to x in (0,10,20 .... ) it is actually running fine ;)

Use this instead query = f"x in {str(tuple([i for i in range(16000) if i % 10 == 0]))}"

[jpambrun-vida]

Thanks. Your formulation is indeed working, even up to deleting 10% of 15,000,000 records (🤯) before memory is an issue. It's a bit hard to understand why it can do 10,000x more like that, but ill take it..

[ion-elgreco]

@jpambrun-vida it could be that datafusion doesn't optimize 5 or 10 or 15 in a better query that can be pushed down