Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Added support to read Avro files' metadata asynchronously #614

Merged
merged 5 commits into from
Nov 19, 2021

Conversation

jorgecarleitao
Copy link
Owner

@jorgecarleitao jorgecarleitao commented Nov 18, 2021

This is one of the two steps necessary to support full async reading for Avro.

Avro is often used as a streaming format and thus supporting async has more benefits than e.g. Parquet.

Supporting async for the remainder of the file will be done on a separate PR.

The main benefit of this PR is that query engines that wish to scan a remote Avro file can do so in an async manner.

As with all other IO modules, the async code:

  • is under a feature gate (io_avro_async)
  • performs minimal CPU work (it still needs to parse a json blob)

@jorgecarleitao jorgecarleitao added the feature A new feature label Nov 18, 2021
@codecov
Copy link

codecov bot commented Nov 18, 2021

Codecov Report

Merging #614 (3903150) into main (9d4107c) will decrease coverage by 0.05%.
The diff coverage is 69.16%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #614      +/-   ##
==========================================
- Coverage   79.61%   79.55%   -0.06%     
==========================================
  Files         403      408       +5     
  Lines       24838    24856      +18     
==========================================
  Hits        19775    19775              
- Misses       5063     5081      +18     
Impacted Files Coverage Δ
src/io/avro/mod.rs 0.00% <0.00%> (ø)
src/io/avro/read_async/mod.rs 0.00% <0.00%> (ø)
src/io/avro/read/mod.rs 81.25% <33.33%> (-1.42%) ⬇️
src/io/avro/read/block.rs 70.58% <70.58%> (ø)
src/io/avro/read/decompress.rs 80.64% <80.64%> (ø)
src/io/avro/read/header.rs 100.00% <100.00%> (ø)
src/io/avro/read/util.rs 95.00% <100.00%> (+9.89%) ⬆️
tests/it/io/avro/read.rs 100.00% <100.00%> (ø)
tests/it/io/avro/read_async.rs 100.00% <100.00%> (ø)
src/io/parquet/read/nested_utils.rs 78.43% <0.00%> (+0.98%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9d4107c...3903150. Read the comment docs.

@jorgecarleitao jorgecarleitao merged commit 3f12bd6 into main Nov 19, 2021
@jorgecarleitao jorgecarleitao deleted the avro_asyn branch November 19, 2021 06:57
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
feature A new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant