Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Start to make Rust node API typed using arrow #353

Merged
merged 35 commits into from
Oct 26, 2023
Merged

Start to make Rust node API typed using arrow #353

merged 35 commits into from
Oct 26, 2023

Conversation

phil-opp
Copy link
Collaborator

@phil-opp phil-opp commented Sep 6, 2023 •
edited
Loading

TODO:

  • update c++ api
  • provide convenience functions for constructing single-value arrays
  • provide convenience functions for downcasting received values to concrete primitives (or even higher-level Rust types)

Open questions:

  • How can we construct arrow arrays from Rust in an ergonomic way?
  • Is zero-copy construction possible?
  • How can we convert received data to higher-level types without copying?

@haixuanTao
Copy link
Collaborator

I think that this looks good in general. It was what I had in mind for a POC.

phil-opp added 14 commits October 16, 2023 12:03
@phil-opp
Merge branch 'main' into rust-typed-input
afae3b7
@phil-opp
Merge branch 'main' into rust-typed-input
b38c112
@phil-opp
Adjust multiple-daemons example
ddf8035
@phil-opp
Simplify arrow array construction through new IntoArrow trait
89f980e
@phil-opp
Simplify reading of primitive and string types from arrow arrays
9a10280 
- introduce a new `ArrayData` wrapper struct
- implement `TryFrom` to convert to primitive types and strings
- update examples to use the new conversion functions
@phil-opp
Add more Rust<->arrow conversions
8f9e2ca
@phil-opp
Update C++ operator API to use new conversions
a54f000
@phil-opp
Fix length call in benchmark example
40b991d
@phil-opp
Rename and reexport Data enums
6140681 
Makes the distinction between raw data and data messages clearer.
@phil-opp
Adjust Python operator API tests
df65eab
@phil-opp
Fix some clippy warnings
631019f
@phil-opp
Fix doc test
14a5531
@phil-opp
Use 'value' instead of 'data' in Python node template
b78f913
@phil-opp
Fix Python operator template too
96eceb8
@haixuanTao haixuanTao mentioned this pull request Oct 19, 2023
Merged
10 tasks
phil-opp added 12 commits October 19, 2023 16:20
@phil-opp
Fix: Remove invalid array length checks when reading byte arrays
10c15d5
@phil-opp
Provide accessor functions for input ID and data in C operator API
36a1f77
@phil-opp
Provide higher-level function to send output from C operator
94c0da6
@phil-opp
Update C dataflow example to use new API functions
75b0513
@phil-opp
Fix name collision
4acc3c6
@phil-opp
Adjust C++ dataflow example for new C operator API
5dcaa9e
@phil-opp
Start fixing Python dataflow example
242513f
@phil-opp
Update arrow and pyo3 dependencies to latest versions
8cc5837
@phil-opp
Rework raw data to arrow array conversion
c6ce457 
- If input has no data, return an empty `NullArray` instead of an empty byte array.
- If input contains Vec data (instead of shared memory data), use safe `arrow::buffer::Buffer::from_vec` constructor, which is also zero-copy.
@phil-opp
Fix alignment errors when receiving empty arrays
ac7d174 
Empty `Vec`s have a dummy pointer that is set to the alignment of the item type. Thus, the pointer for empty `Vec<u8>` data is `0x1`. Using this pointer to construct a `ArrayData` instance can lead to alignment errors if the arrow array is of a type with larger alignment, e.g. float64.

This commit fixes the alignment error by checking for an empty raw buffer and constructing an empty `ArrayData` instance in this case.

Fixes #362
@phil-opp
Avoid alignment errors by aligning raw data on deseralization
ad3f048 
Ensures that all raw data has at least an alignment of 128.
@phil-opp
Add more context to serialize_deserialize roundrip errors
905e9ee
@phil-opp phil-opp force-pushed the rust-typed-input branch 2 times, most recently from 6c90a54 to d2c8239 Compare October 25, 2023 10:45
@phil-opp phil-opp marked this pull request as ready for review October 25, 2023 12:23
@phil-opp phil-opp requested a review from haixuanTao October 25, 2023 12:23
haixuanTao
haixuanTao approved these changes Oct 26, 2023
View reviewed changes
Copy link
Collaborator

@haixuanTao haixuanTao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect. Works very well thanks!

@phil-opp
Copy link
Collaborator Author

Great, thanks for reviewing!

@phil-opp phil-opp merged commit 0ef994e into main Oct 26, 2023
@phil-opp phil-opp deleted the rust-typed-input branch October 26, 2023 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@haixuanTao haixuanTao haixuanTao approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@phil-opp @haixuanTao