Skip to content

Commit

Permalink
allow using custom datetime format for inference and parsing csv file (
Browse files Browse the repository at this point in the history
…#1112)

* allow using custom datetime format for inference and parsing csv file

The patch extends the current implementation to allow passing a custom
datetime_re and datetime_format to the ReaderBuilder.

datetime_re is used infer schema of the csv and then datetime_format is
used to parse the actual string to a Date64.
ofcourse  passing non-compatible datetime_re and datetime_format values
is going to fail the parsing or inference, however it is an expected but
hard-to-detect failure.

* Incorporate some clippy recommendations for limit count of call args

The patch adds a new struct to collect all these options together and
then passes the struct around. Ideally the struct could be embedded into
the reader but that can be done as separate exercise.

* Detect presence of timezone in format while parsing csv for date64

The patch decides on using NaiveDateTime or DateTime from chrono lib
based on presence of timezone components

chrono expects timezone to be presetn if DateTime is used, errors
otherwise. Whereas NaiveDateTime ignores timezone even if explicitly
provided.
  • Loading branch information
sum12 authored Jan 2, 2022
1 parent 89d5273 commit d852229
Show file tree
Hide file tree
Showing 3 changed files with 280 additions and 131 deletions.
2 changes: 1 addition & 1 deletion arrow/examples/read_csv.rs
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ fn main() {
let file = File::open("test/data/uk_cities.csv").unwrap();

let mut csv =
csv::Reader::new(file, Arc::new(schema), false, None, 1024, None, None);
csv::Reader::new(file, Arc::new(schema), false, None, 1024, None, None, None);
let _batch = csv.next().unwrap().unwrap();
#[cfg(feature = "prettyprint")]
{
Expand Down
Loading

0 comments on commit d852229

Please sign in to comment.