Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compatability with arrow fixed-shape-tensor extension #3004

Open
jleibs opened this issue Aug 16, 2023 · 0 comments
Open

Compatability with arrow fixed-shape-tensor extension #3004

jleibs opened this issue Aug 16, 2023 · 0 comments
Labels
🏹 arrow concerning arrow codegen/idl enhancement New feature or request

Comments

@jleibs
Copy link
Member

jleibs commented Aug 16, 2023

The arrow project has an official extension for fixed-shape-tensors: https://arrow.apache.org/docs/dev/format/CanonicalExtensions.html#fixed-shape-tensor

Our current tensor APIs use a rerun-specific tensor format that stores tensor-dimension as a data field, giving us support for dynamic-shape-tensor.

Using fixed-shape-tensor ourselves would be quite a bit of work at this point with two major challenges:

  1. We need a mechanism to surface metadata from components.
  2. We need to be careful about how metadata is tracked and impacted by things like batching.

That said, users that are already working with arrow tensors from some other library would benefit from ergonomic support for forwarding these on into Rerun directly. This might be another interesting use-case for derived components.

@jleibs jleibs added enhancement New feature or request 🏹 arrow concerning arrow codegen/idl labels Aug 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏹 arrow concerning arrow codegen/idl enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant