You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What happened: Merge throws an error when there is an arrow datatype decimal128 present in any of the columns. Small numbers e.g. 35 work. 0 doesn't work, 10**20 doesn't work
What you expected to happen: Execute a merge
How to reproduce it:
import pyarrow as pa
import deltalake as dl
from deltalake import DeltaTable
from pathlib import Path
breaking_number = 0 # also doesnt work with something large like 10**20
# Simplified schema with only one column
schema = pa.schema([("decimalcol", pa.decimal128(38, 0))
, ("key", pa.int32())])
# Create dummy data
data = [{'decimalcol': breaking_number, 'key': 1}]
table = pa.Table.from_pylist(data, schema=schema)
# Define file path
path = Path('testdir')
dl.write_deltalake(path, table, mode="overwrite")
table = pa.Table.from_pylist(data, schema=schema)
existing_table = DeltaTable(path)
(
existing_table.merge(
source=table,
predicate= "s.key = t.key",
source_alias="s",
target_alias="t") # would require a predicate (primary key) to merge
.when_matched_update_all()
.when_not_matched_insert_all()
.execute()
)
More details:
The text was updated successfully, but these errors were encountered:
@Tommel71 it's related to this issue: #2193, so closing in favor of that one. Which is actually on hold since we are dependent on seeing improved value parsing support in arrow-rs
Environment
Delta-rs version:
deltalake==0.16.0
pyarrow==11.0.0
Binding: Python
Environment: Ubuntu 22.04.4 LTS
Bug
What happened: Merge throws an error when there is an arrow datatype decimal128 present in any of the columns. Small numbers e.g. 35 work. 0 doesn't work, 10**20 doesn't work
What you expected to happen: Execute a merge
How to reproduce it:
More details:
The text was updated successfully, but these errors were encountered: