Make DataFusion Core compile #3
Conversation
| @@ -0,0 +1,302 @@ | |||
| // Licensed to the Apache Software Foundation (ASF) under one | |||
There was a problem hiding this comment.
This file is a copy of arrow/src/compute/kernels/cast_utils.rs.
Arrow2 currently only provide utf8_to_timestamp_ns_scalar with explicit timezone and fmt string, or utf8_to_naive_timestamp_ns_scalar with explicit fmt string. It's not possible to achieve identical semantic with the original arrow-rs string_to_timestamp_nanos in arrow2 since (as I quote jorge here)
it mixes timezone-aware, non-timezone aware, and local time in the same conversion. That IMO is not a “cast”: it is a “let me try anything to not crash irrespectively of its correctness”. For people that do rely on timezones, imo that is a foot gun that hides bugs. It also goes against the arrow spec that states that naive timestamps are not utc.
I added this back as a temporary workaround to make the build pass. We need to redesign the time-related API by providing users with two possible flavors: one with explicit timezone setting and the other default to local timezone, but both of the API variants should ask user for datetime format string. I'll file an issue on this later.
There was a problem hiding this comment.
i think it's the right decision to move this logic into datafusion.
| @@ -197,7 +197,7 @@ impl ExecutionPlan for SortPreservingMergeExec { | |||
| /// `SortKeyCursor::compare` can then be used to compare the sort key pointed to | |||
| /// by this row cursor, with that of another `SortKeyCursor`. A cursor stores | |||
| /// a row comparator for each other cursor that it is compared to. | |||
| struct SortKeyCursor<'a> { | |||
| struct SortKeyCursor { | |||
There was a problem hiding this comment.
I ported apache/arrow-rs#542 back to arrow2, please merge the latest master into your ord branch. @houqp
|
@houqp To make your local I have this in my root cargo.toml now: [patch.crates-io]
arrow2 = { path = "/Users/shenyijie/oss/arrow2" }
parquet2 = { path = "/Users/shenyijie/oss/parquet2"} |
* # This is a combination of 3 commits. # This is the 1st commit message: Add Display for Expr::BinaryExpr # This is the commit message #2: Update logical_plan/operators tests # This is the commit message #3: rebase and debug display for non binary expr * Add Display for Expr::BinaryExpr Update logical_plan/operators tests rebase and debug display for non binary expr Add Display for Expr::BinaryExpr Update logical_plan/operators tests Updating tests Update aggregate display Updating tests without aggregate More tests Working on agg/scalar functions Fix binary_expr in create_name function and attendant tests More tests More tests Doc tests Rebase and update new tests * Submodule update * Restore submodule references from master Co-authored-by: Andrew Lamb <[email protected]>
No description provided.