Support for Decimal series?

[j-a-m-l]

Since Arrow2 provides support for the Decimal data type, is there any plan to allow it for Polars series?

[pengyizeng]

x2

[gussen]

I am not familiar with the arrow format and the inner workings of polars but I guess that adding proper support for decimal is a significant work. In the mean time a probably easier thing to do is convert decimal with scale 0, precision 1-9 to Int32, decimal with scale 0, precision 10-19 to Int64 and in all other cases convert to Text. Polars can then read the files with decimal and do a lot of analysis with the functionality that exists for Int and Text and of course do analysis on all the other fields.

[thomasaarholt]

At the least, it would be nice if pl.read_parquet automatically cast from the decimal type to float type, like pl.from_arrow seems to do. Instead of just throwing an error:

one_row.zip

import pyarrow.parquet
data = pyarrow.parquet.read_table("one_row.parquet")
data["pred_mean"].type # Decimal128Type(decimal128(38, 0))
df = pl.from_arrow(data) 
df["pred_mean"].dtype # polars.datatypes.Float64


df = pl.read_parquet("one_row.parquet")
# InvalidOperationError: Cannot create polars series from Decimal(38, 0) type

[WillenCrypto]

Same issue here. Please support decimal!

[sa-]

I was trying out polar's new delta reader and ran into this error using pl.scan_delta(...)

Arrow datatype Decimal(18, 5) not supported by Polars. You probably need to activate that data-type feature

I've installed polars with pip install polars[all] so it's unlikely that the feature is missing. It would be nice if polars could support this.

[rossmechanic]

+1!

[houyingkun]

+1!

[plaflamme]

Is there a rundown of what it would take to add decimal support to pola-rs? A previous comment mentioned this might be a large effort, but perhaps if it's broken down into manageable pieces, the community could take this on?

I understand that breaking this down is also a bunch of work in itself, so I totally understand that this isn't done nor prioritized. Still, providing some guidance to new contributors might be a way to get this done at a lower bar than doing the work itself.

Also, as a side-note, I would recommend making the conversion to f64 require being explicitly enabled by the user, e.g.: pl.read_parquet("foo.parquet", decimal_as_f64 = True) and fail when not enabled. I think it's surprising that pola-rs does this by default (logging isn't a bad idea, but it can easily be missed in some deployment scenarios) since it can lead to issues if the computations need to be exact (money or otherwise).

[rossmechanic]

Agreed with @plaflamme. I'd be happy to contribute to some of the work to support decimal type. It's the only thing keeping us from moving from pandas to polars.

[ritchie46]

I can see of I can take some time for this in the coming weeks.

[plaflamme]

@ritchie46 That's great news! Please note that I'd be happy to help out this effort if there's a way to break it down. I can also help testing.

[ritchie46]

@ritchie46 That's great news! Please note that I'd be happy to help out this effort if there's a way to break it down. I can also help testing.

That would be great. The first step would be adding ChunkedArray<Int128Type>. This would be adding that logic to the existing Numeric traits and types. This should be added under a dtype-i128 feature flag.

We could merge this first and then we'll have to look what the best next steps are.

[plaflamme]

Alright, here's a draft PR, feedback would be greatly appreciated.

[arachnegl]

I believe Polars supports Decimals since https://github.com/pola-rs/polars/releases/tag/py-0.16.10

I have added #7566 relating to the current implementation.

[martin-wiebusch-thg]

In polars[all] v0.16.13 I still get the following when trying to load a parquet file with decimals in it:

Arrow datatype Decimal(38, 9) not supported by Polars. You probably need to activate that data-type feature.

How do I activate the feature?

[naarkhoo]

I got the same with '0.16.13' when reading with scan_parquet
thread '<unnamed>' panicked at 'Arrow datatype Decimal(2, 1) not supported by Polars. You probably need to activate that data-type feature.', /Users/runner/work/polars/polars/polars/polars-core/src/datatypes/field.rs:153:19

[alexander-beedie]

How do I activate the feature?

@martin-wiebusch-thg, @naarkhoo:

pl.Config.activate_decimals()

Still in early development/testing, hence the need to explicitly activate (for now).

[alexander-beedie]

Now that the feature exists (in an early/development state), I'm closing this issue in favour of more specific issues/requirements found with the implementation...👍