ARROW2: support avro

[houqp]

Avro integration needs to be fixed.

[Igosuki]

Is this about the dict issue or is it something else ?

[houqp]

This is something else, we need to port your avro implementation to https://github.com/jorgecarleitao/arrow2/tree/main/src/io/avro for #68 ;)

[houqp]

@Igosuki if you are interested in taking on this work, please coordinate with @jorgecarleitao .

[Igosuki]

@houqp will do

[jorgecarleitao]

The approach I took in arrow2 was to not use AvroValue and work directly with the byte stream. The reason for this is that AvroValue takes bytes by value and not by reference, which implies that when we perform the conversion File -> AvroValue -> Arrow for a Utf8 or Binary, we end up with the transformation bytes -> AvroValue::String(String) -> Utf8Array, performing an extra allocation per item.

Since the avro format is relatively simple to read from, I just implemented a reader from bytes directly to arrow. I did not implemented it for all types, only for the basic ones, but the idea stands.

So, if I understood, the goal is to generalize the parser to more types. Which ones are needed here?

[houqp]

I think struct and list are the main ones: https://github.com/apache/arrow-datafusion/blob/5cc4e9f53fab29e81ea7c98baac8ce277a0cb54a/datafusion/src/avro_to_arrow/arrow_array_reader.rs#L530

[Igosuki]

There is enum, array, dictionary (map), and logical types, these are almost the same as parquet, as parquet a was meant to be a columnar version of Avro. Le mer. 6 oct. 2021 à 07:02, QP Hou ***@***.***> a écrit :

…

I think struct and list are the main ones: https://github.com/apache/arrow-datafusion/blob/5cc4e9f53fab29e81ea7c98baac8ce277a0cb54a/datafusion/src/avro_to_arrow/arrow_array_reader.rs#L530 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1061 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADDFBX6BDS7EBDQATLEVDDUFPJ45ANCNFSM5E4QL77A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[jorgecarleitao]

as parquet a was meant to be a columnar version of Avro

ahhh, now it makes sense why they are so similar! Thanks a lot for sharing that.

I've implemented the remaining types here: jorgecarleitao/arrow2#493 . We are missing Struct and Decimal.

[Igosuki]

@jorgecarleitao Nice one, seems like you got the encoding part almost rounded up

[houqp]

@Igosuki I believe you have completed the migration for all avro related code right? Is there any left over for this issue?

[Igosuki]

No, besides the patch I mentioned which enables arrays to be null https://github.com/jorgecarleitao/arrow2/blob/main/src/io/avro/read/deserialize.rs#L101

[houqp]

Thanks @Igosuki for all your work on Avro :)