Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error when reading delta table with IDENTITY column #2152

Closed
Ghodot opened this issue Jan 31, 2024 · 0 comments
Closed

Error when reading delta table with IDENTITY column #2152

Ghodot opened this issue Jan 31, 2024 · 0 comments
Assignees
Labels
bug Something isn't working
Milestone

Comments

@Ghodot
Copy link

Ghodot commented Jan 31, 2024

Environment

Delta-rs version: 0.15.1

Binding: Python 3.8

Environment: Local

  • Cloud provider: Azure
  • OS: Linux
  • Other: External table mounted on adls gen 2 mount

Bug

What happened:
I'm trying to read an external table containing an IDENTITY column. However I encounter the following error when running locally :

    dt = DeltaTable(path)
  File "/home/***/Documents/***/venv/lib/python3.8/site-packages/deltalake/table.py", line 396, in __init__
    self._table = RawDeltaTable(
_internal.DeltaError: Delta protocol violation: Kernel: Invalid url: data did not match any variant of untagged enum MetadataValue at line 1 column 181

If I recreate the table without the Identity column, it works correctly

What you expected to happen:
The table is read correctly in python

How to reproduce it:
Create a table with an identity :

CREATE TABLE spark_catalog.data_model.d_dates (
  ID_D_DATE BIGINT GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1),
  TXT_DateKey STRING
) USING delta LOCATION 'dbfs:/mnt/adls_gen2_mount/d_dates' TBLPROPERTIES ('delta.minReaderVersion' = '1','delta.minWriterVersion' = '3')

Mount the adls endpoint locally or copy the folder to your local computer.
Read the folder using DeltaTable(local_path)

More details:
I tried changing the minWriterVersion / switching between GENERATED ALWAYS and GENERATED BY DEFAULT but none worked.
Except for local access, the table works just fine when working with SQL or spark.

@Ghodot Ghodot added the bug Something isn't working label Jan 31, 2024
@rtyler rtyler self-assigned this Feb 1, 2024
@rtyler rtyler added this to the Rust v0.17 milestone Feb 1, 2024
rtyler added a commit to rtyler/delta-rs that referenced this issue Feb 1, 2024
rtyler added a commit to rtyler/delta-rs that referenced this issue Feb 1, 2024
@rtyler rtyler closed this as completed in bdb03a3 Feb 1, 2024
RobinLin666 pushed a commit to RobinLin666/delta-rs that referenced this issue Feb 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants