SchemaError occurs during table optimisation after upgrade to v0.18.1

[r1fad]

Environment

Delta-rs version: v0.18.1

Binding: rust

---

Bug

What happened: In my project, there is a service that runs on a scheduled basis to compact and optimise delta tables. After upgrading delta-rs to v0.18.1 (from v0.17.1) we see an error saying Delta(Arrow { source: SchemaError("Could not find column keys") }). There is no column called 'keys' in our datasets so I am do not understand why we get this error. The error comes from deltalake-core-0.18.1/src/operations/cast.rs:150.

What you expected to happen:
No schema errors should be reported when upgrading to a new version of delta-rs.

How to reproduce it:
Not entirely sure how to reproduce it since we do not have a column called 'keys' in our datasets.

More details:
This error does not occur in the version we run in production which is v0.17.1 (commit hash 25962a0). The look back window on the compactor service is 2 days. So after 2 days of running the new version, we no longer see the SchemaError which leads me to believe that the v0.18.1 cannot optimise tables that contain data that was inserted by v0.17.1.

[r1fad]

@ion-elgreco how can I incorporate your fix into my project?

[ion-elgreco]

@ion-elgreco how can I incorporate your fix into my project?

You will have to wait for the next release when we bump kernel

[r1fad]

@ion-elgreco any idea on when the next release of delta-rs is coming? I imagine this fix is important for quite a lot of people. And the change introduced earlier with the kernel is a breaking change.

I also noticed that after downgrading my project to v0.17.1, it can no longer optimise tables that contain data inserted by v0.18.1. This issue should disappear after 2 days though because lookback window is 2 days

[ion-elgreco]

@r1fad somewhere today or this weekend it will be released