Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set adjusted to UTC if UTC timezone (#1932) #1937

Merged
merged 1 commit into from
Jun 24, 2022
Merged

Set adjusted to UTC if UTC timezone (#1932) #1937

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 37 additions & 2 deletions parquet/src/arrow/schema.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,10 +300,15 @@ fn arrow_to_parquet_type(field: &Field) -> Result<Type> {
.with_repetition(repetition)
.build()
}
DataType::Timestamp(time_unit, _) => {
DataType::Timestamp(time_unit, tz) => {
let is_utc = tz
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I initially played around with using chrono-tz, but this is a non-trivial additional dependency, and does not appear to be being actively maintained. I think this should be good enough, until such a time as we potentially add coerce types functionality as part of #1666

.as_ref()
.map(|tz| tz == "UTC" || tz == "+00:00" || tz == "-00:00")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically GMT is the timezone - https://www.timeanddate.com/time/gmt-utc-time.html
Should GMT be added in the checks too ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I would rather keep this small and simple, rather than cover all timezones that happen to be equivalent. Longer term this may be handled as part of #1938

.unwrap_or(false);

Type::primitive_type_builder(name, PhysicalType::INT64)
.with_logical_type(Some(LogicalType::Timestamp {
is_adjusted_to_u_t_c: false,
is_adjusted_to_u_t_c: is_utc,
unit: match time_unit {
TimeUnit::Second => unreachable!(),
TimeUnit::Millisecond => {
Expand Down Expand Up @@ -1281,6 +1286,11 @@ mod tests {
OPTIONAL INT64 time_micro (TIME_MICROS);
OPTIONAL INT64 ts_milli (TIMESTAMP_MILLIS);
REQUIRED INT64 ts_micro (TIMESTAMP(MICROS,false));
REQUIRED INT64 ts_seconds;
REQUIRED INT64 ts_micro_utc (TIMESTAMP(MICROS, true));
REQUIRED INT64 ts_millis_zero_offset (TIMESTAMP(MILLIS, true));
REQUIRED INT64 ts_millis_zero_negative_offset (TIMESTAMP(MILLIS, true));
REQUIRED INT64 ts_micro_non_utc (TIMESTAMP(MICROS, false));
REQUIRED GROUP struct {
REQUIRED BOOLEAN bools;
REQUIRED INT32 uint32 (INTEGER(32,false));
Expand Down Expand Up @@ -1329,6 +1339,31 @@ mod tests {
DataType::Timestamp(TimeUnit::Microsecond, None),
false,
),
Field::new(
"ts_seconds",
DataType::Timestamp(TimeUnit::Second, Some("UTC".to_string())),
false,
),
Field::new(
"ts_micro_utc",
DataType::Timestamp(TimeUnit::Microsecond, Some("UTC".to_string())),
false,
),
Field::new(
"ts_millis_zero_offset",
DataType::Timestamp(TimeUnit::Millisecond, Some("+00:00".to_string())),
false,
),
Field::new(
"ts_millis_zero_negative_offset",
DataType::Timestamp(TimeUnit::Millisecond, Some("-00:00".to_string())),
false,
),
Field::new(
"ts_micro_non_utc",
DataType::Timestamp(TimeUnit::Microsecond, Some("+01:00".to_string())),
false,
),
Field::new(
"struct",
DataType::Struct(vec![
Expand Down