v0.3.0

This is the first Rust implementation to support reading parquet on wasm, which opens up a whole new range of possibilities by allowing reading parquet directly on a browser without having to communicate data to a server.

This is also the first time an implementation of arrow in Rust offers native async support to read parquet, enabling consumers to perform ranged queries against blob storage without blocking. Check out an example here

Finally, this implementation now offers a Scalar API that we can build upon to support arrow's more exotic types such as maps and unions.

Thanks a lot to @sundy-li , @PsiACE , @ritchie46, @ghuls and @Dandandan for the contributions and discussions and to @Darksonn for all the patience and help that unblocked me when working on the async parquet support.

Full Changelog

Breaking changes:

• Renamed sum to sum_primitive #273
• Moved trait Index from array::Index to types::Index #272
• Added optional projection to IPC FileReader #271
• Added optional page_filter to parquet's RecordReader and get_page_iterator #270
• Renamed parquets' CompressionCodec to Compression #269

New features:

• Added support for FFI of dictionary-encoded arrays #267 (jorgecarleitao)
• Added support for projection pushdown on IPC files #264 (jorgecarleitao)
• Added support to read parquet asynchronously #260 (jorgecarleitao)
• Added support to filter parquet pages. #256 (jorgecarleitao)
• Added wrapping_cast to cast kernels #254 (sundy-li)
• Added support to parquet IO on wasm32 #239 (jorgecarleitao)
• Added support to round-trip dictionary arrays on parquet #232 (jorgecarleitao)
• Added Scalar API #56 (jorgecarleitao)

Fixed bugs:

• Fixed error in computing remainder of chunk iterator #262 (jorgecarleitao)
• Fixed error in slicing bitmap. #250 (jorgecarleitao)

Enhancements:

• Improve the performance in cast kernel using AsPrimitive trait in generic dispatch #252
• Poor performance in sort::sort_to_indices with limit option in arrow2 #245
• Support loading Feather v2 (IPC) files with more than 1 million tables #231
• Migrated to parquet2 v0.3 #265 (jorgecarleitao)
• Added more tests to cast and min/max #253 (jorgecarleitao)
• Prettytable is unmaintained. Change to comfy-table #251 (PsiACE)
• Added IndexRange to remove checks in hot loops #247 (jorgecarleitao)
• Make merge_sort_slices MergeSortSlices public #243 (sundy-li)

Documentation updates:

• Added example and guide section on compute #242 (jorgecarleitao)

Closed issues:

• Allow projection pushdown to IPC files #261
• Add support to write dictionary-encoded pages #211
• Make IpcWriteOptions easier to find. #120

v0.2.0

Changelog

v0.2.0 (2021-07-30)

Full Changelog

Breaking changes:

• Simplified new signature of growable API #238 (jorgecarleitao)
• Add support to merge sort with a limit #222 (sundy-li)
• Generalized sort to accept indices other than i32. #220 (jorgecarleitao)
• Added support for limited sort #218 (jorgecarleitao)

New features:

• Merge sort support limit option #221
• Introduce limit option to sort #215
• Added support for take of interval of days_ms #219 (jorgecarleitao)
• Added FFI for remaining types #213 (jorgecarleitao)

Fixed bugs:

• Filter operation on sliced utf8 arrays are incorrect #233
• Fixed error in slicing bitmap. #237 (jorgecarleitao)
• Fixed nested FFI. #212 (jorgecarleitao)

Enhancements:

• Avoid materialization of indices in filter_record_batch for single arrays #234
• Add integration tests for writing to parquet #80
• Short-circuited boolean evaluation in GrowableList #228 (ritchie46)
• Add extra inlining to speed up take #226 (Dandandan)
• Removed un-needed unsafe #225 (jorgecarleitao)

Documentation updates:

• Add documentation to guide #96
• Add git submodule command to correct the test doc #223 (sundy-li)
• Added badges to README #216 (sundy-li)
• Clarified differences with arrow crate #210 (alamb)
• Clarified differences with arrow crate #209 (alamb)

* This Changelog was automatically generated by github_changelog_generator