SnowflakeArrow

Snowflake specific Arrow implementation, which only implements the Data types Snowflake supports.

This has been designed to work with req_snowflake, which when added to a project with req_snowflake will allow Snowflake to send back Arrow results which this library will parse.

This library uses Rustler to create a Rust binding to Arrow2 library.

Performance has been given a priority, so some workarounds have had to be made to get good performance, as we are returning datasets.

Data is returned from Polars, which is what Explorer uses.

Installation

def deps do
  [
    {:snowflake_arrow, github: "joshuataylor/snowflake_arrow"}
  ]
end

Features

• Cast to DateTime & Date from within Rust.
• Custom Snowflake types (TIMESTAMP_TZ, TIMESTAMP_NTZ, TIMESTAMP_LTZ) supported.
• Optimised for speed

Snowflake Notes

Snowflake send back Timestamps in Structs which include the timezone. This is casted to their proper types during import, which Polars makes incredibly easy.

Benchmarks

Benchmarks are hard to compare against the JSON implementation of Snowflake, as the JSON implementation doesn't return the same results as the Arrow implementation.