Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge breaks using logical datatype decimal128 #2343

Closed
Tommel71 opened this issue Mar 26, 2024 · 1 comment
Closed

Merge breaks using logical datatype decimal128 #2343

Tommel71 opened this issue Mar 26, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@Tommel71
Copy link

Environment

Delta-rs version:
deltalake==0.16.0
pyarrow==11.0.0

Binding: Python

Environment: Ubuntu 22.04.4 LTS


Bug

What happened: Merge throws an error when there is an arrow datatype decimal128 present in any of the columns. Small numbers e.g. 35 work. 0 doesn't work, 10**20 doesn't work

What you expected to happen: Execute a merge

How to reproduce it:

import pyarrow as pa
import deltalake as dl
from deltalake import DeltaTable
from pathlib import Path

breaking_number = 0 # also doesnt work with something large like 10**20

# Simplified schema with only one column
schema = pa.schema([("decimalcol", pa.decimal128(38, 0))
                    , ("key", pa.int32())])
# Create dummy data
data = [{'decimalcol': breaking_number, 'key': 1}]
table = pa.Table.from_pylist(data, schema=schema)
# Define file path
path = Path('testdir')

dl.write_deltalake(path, table, mode="overwrite")


table = pa.Table.from_pylist(data, schema=schema)

existing_table = DeltaTable(path)

(
    existing_table.merge(
        source=table,
        predicate= "s.key = t.key",
        source_alias="s",
        target_alias="t")  # would require a predicate (primary key) to merge
    .when_matched_update_all()
    .when_not_matched_insert_all()
    .execute()
)

More details:

@Tommel71 Tommel71 added the bug Something isn't working label Mar 26, 2024
@ion-elgreco
Copy link
Collaborator

ion-elgreco commented Mar 28, 2024

@Tommel71 it's related to this issue: #2193, so closing in favor of that one. Which is actually on hold since we are dependent on seeing improved value parsing support in arrow-rs

@ion-elgreco ion-elgreco closed this as not planned Won't fix, can't repro, duplicate, stale Mar 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants