Issue writing to mounted storage in AKS using delta-rs library

[tcharbonnier]

Environment

Azure AKS version 1.28.5, mounted ADLS Gen 2 Storage in Azure, running delta-rs from a pod.

Delta-rs version:
v0.16.4

Binding:

Environment:

• Cloud provider: Azure
• OS: Ubuntu
• Other: Python3 version 3.12.2

bcrypt==4.1.2
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
cryptography==42.0.5
deltalake==0.16.4
idna==3.6
numpy==1.26.4
pandas==2.2.1
paramiko==3.4.0
pyarrow==15.0.2
pyarrow-hotfix==0.6
pycparser==2.22
PyNaCl==1.5.0
pysftp==0.2.9
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
pytz==2024.1
requests==2.31.0
setuptools==68.1.2
six==1.16.0
tzdata==2024.1
urllib3==2.2.1
wheel==0.42.0

---

Bug

An error is returned when attempting to write to mounted storage. Please note this mounted storage works without failure when writing/reading manually, its only when running code from this library we get the error.

What happened:

OSError: Generic LocalFileSystem error: Unable to copy file from /mnt/storage-main/delta/services/_delta_log/_commit_67f14f8d-e125-4d76-9704-7f5a5b991fe5.json.tmp to /mnt/storage-main/delta/services/_delta_log/00000000000000000000.json: Function not implemented (os error 38)

What you expected to happen:

Code should copy data to the mounted storage account. but fails with the above error.
How to reproduce it:

mount storage via csi driver to a pod in aks, use library to write data back to the storage account.
More details:

[ion-elgreco]

@tcharbonnier you should pass this to the storage options (this disables safe writes, meaning if you have concurrent writers, you could overwrite each other):

    storage_options= {
        "allow_unsafe_rename": "true",
    }

I also highly suggest to write directly to ADLS so that you do get concurrent write support