Parquet TZ aware datetimes not correctly read by the default engine

[FlorianGD]

What language are you using?

Python

What version of polars are you using?

0.13.4

What operating system are you using polars on?

Ubuntu 20.04

What language version are you using

python 3.8.12

Describe your bug.

Parquet files with time zones are not accurately read by the default engine. Using pyarrow is fine, but slower.

What are the steps to reproduce the behavior?

import pandas as pd
import polars as pl
df = pd.DataFrame(data = {"Timestamp": pd.date_range("2022-01-01T00:00+00:00", "2022-01-01T10:00+00:00", freq="H")})
#                    Timestamp
# 0  2022-01-01 00:00:00+00:00
# 1  2022-01-01 01:00:00+00:00
# 2  2022-01-01 02:00:00+00:00
# 3  2022-01-01 03:00:00+00:00
# 4  2022-01-01 04:00:00+00:00
# 5  2022-01-01 05:00:00+00:00
# 6  2022-01-01 06:00:00+00:00
# 7  2022-01-01 07:00:00+00:00
# 8  2022-01-01 08:00:00+00:00
# 9  2022-01-01 09:00:00+00:00
# 10 2022-01-01 10:00:00+00:00
file = "/tmp/test.parquet"
df.to_parquet(file)
pd.read_parquet(file)  # File is read fine with timezone info kept
#                    Timestamp
# 0  2022-01-01 00:00:00+00:00
# 1  2022-01-01 01:00:00+00:00
# 2  2022-01-01 02:00:00+00:00
# 3  2022-01-01 03:00:00+00:00
# 4  2022-01-01 04:00:00+00:00
# 5  2022-01-01 05:00:00+00:00
# 6  2022-01-01 06:00:00+00:00
# 7  2022-01-01 07:00:00+00:00
# 8  2022-01-01 08:00:00+00:00
# 9  2022-01-01 09:00:00+00:00
# 10 2022-01-01 10:00:00+00:00
pl.read_parquet(file)  # Timezone is lost, but also the offset
# Conversion of timezone aware to naive datetimes. TZ information may be lost.
# shape: (11, 1)
# ┌─────────────────────────┐
# │ Timestamp               │
# │ ---                     │
# │ datetime[ns]            │
# ╞═════════════════════════╡
# │ 1970-01-19 23:49:55.200 │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ 1970-01-19 23:49:58.800 │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ 1970-01-19 23:50:02.400 │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ 1970-01-19 23:50:06     │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ ...                     │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ 1970-01-19 23:50:20.400 │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ 1970-01-19 23:50:24     │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ 1970-01-19 23:50:27.600 │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ 1970-01-19 23:50:31.200 │
# └─────────────────────────┘
pl.read_parquet(file, use_pyarrow=True)  # Now the offset is correct, but TZ info is lost
# Conversion of timezone aware to naive datetimes. TZ information may be lost.
# shape: (11, 1)
# ┌─────────────────────┐
# │ Timestamp           │
# │ ---                 │
# │ datetime[μs]        │
# ╞═════════════════════╡
# │ 2022-01-01 00:00:00 │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ 2022-01-01 01:00:00 │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ 2022-01-01 02:00:00 │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ 2022-01-01 03:00:00 │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ ...                 │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ 2022-01-01 07:00:00 │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ 2022-01-01 08:00:00 │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ 2022-01-01 09:00:00 │
# ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
# │ 2022-01-01 10:00:00 │
# └─────────────────────┘

Note that this happens only when the datetime has a timezone, if it is naïve, the datetime is correctly read.

What is the expected behavior?

As the default engine is faster, I would have liked it to read at least the correct offset.

Also, I did not manage to find an argument that I could pass so that the TZ is retained, maybe I missed something in the documentation.

Thank you for this project, I like it a lot, and I'll try to onboard my colleagues on it :)

[ritchie46]

Polars does not yet support timezones.

[FlorianGD]

Oh, OK, I did not know. But even without the TZ, would it be possible to have the correct offset for the dates?

[ritchie46]

But even without the TZ, would it be possible to have the correct offset for the dates?

Yep, will look at that.

[ritchie46]

jorgecarleitao/arrow2#861