Release GIL in deltalake.write_deltalake

[mrocklin]

I'm running deltalake.write_deltalake many times in parallel and noticing that my Python process is freezing up a bit. I suspect that this function has a long period where it doesn't release the GIL. Is this true?

If so, I suspect that it's likely accidental, and could be easily changed. I'm not familiar with the rust-Python tooling, but most Python binding systems make this pretty easy.

[ion-elgreco]

@mrocklin which engine are you using to write with?

With the rust engine I don't think we are releasing the GIL but shouldn't be to tricky to add. It's there for MERGE

[mrocklin]

It looks like this maybe also happens for compact (and presumably other operations)

[mrocklin]

Whatever's default. My code looks like this:

    deltalake.write_deltalake(
        outfile,
        df,
        mode="append",
        storage_options=STORAGE_OPTIONS,
        partition_by="date",
    )

[ion-elgreco]

The default uses the PyArrow engine.

You could try compiling a version where the called method in rust is wrapped inside py.allow_threads, feel free to open a PR :)

[mrocklin]

Alas I'm not familiar with the Rust compilation/build process, so it's unlikely that I'll do this work myself (hopefully it's ok to raise an issue without volunteering to do the work myself).

[franz101]

mhh I tried:

  deltalake.write_deltalake(
        outfile,
        df,
        mode="append",
        storage_options=STORAGE_OPTIONS,
        partition_by="date",
        engine="rust"
    )

so you suggest to add this line
m.add_function(py.allow_threads(|| [search_sequential(contents, needle)](pyo3::wrap_pyfunction!(write_to_deltalake, m)?)))?;
to lib.rs correct?

[ion-elgreco]

@franz101 you can do the same thing as done for merge:

https://github.com/delta-io/delta-rs/blob/main/python%2Fsrc%2Flib.rs#L520-L541

[franz101]

Ah nice, is there any benefit of making it up to the user if the function should block GIL or not?

[mrocklin]

Ah nice, is there any benefit of making it up to the user if the function should block GIL or not?

Almost certainly not is my guess. If the code genuinely creates/destroys Python objects then it should not release the GIL. If it doesn't screw around with Python objects at all (my hope) then it should definitely release the GIL.

FWIW I don't know of any other project that makes GIL-holding optional. It's either a good idea or it isn't, entirely dependent on the wrapped code.

[mrocklin]

(also, thank you for looking into this!)

[franz101]

(also, thank you for looking into this!)
Thank you for the one billion challenge (though the trend goes to one trillion now).

I opened the PR tests are passing fine.
#2257

My next PR if the demand is there to use custom PK SK for DynamoDB since for DynamoDB latency is lower if using an existing database

[mrocklin]

Thank you @franz101 and @ion-elgreco !

Thank you for the one billion challenge (though the trend goes to one trillion now).

Ha, yes, the 1TRC is fun. A non-trivial amount of time is listing the parquet files, so adding delta into the mix would probably shave off a non-trivial amount of time :)

Should I raise another issue for the functions other than write_deltalake?