Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Added support for reading IPC with more than 1 million columns #240

Merged
merged 1 commit into from
Jul 31, 2021
Merged

Added support for reading IPC with more than 1 million columns #240

merged 1 commit into from
Jul 31, 2021

Conversation

ghuls
Copy link
Contributor

@ghuls ghuls commented Jul 30, 2021 •
edited by jorgecarleitao
Loading

Set Flatbuffer verification parameters to the same settings as the
C++ arrow implementation (ARROW-11559). This change allows reading
IPC data with more than 1 milion columns.

Closes: #231

@ghuls
Set Flatbuffer verification parameters to the same settings as the C+…
771fd2a 
…+ arrow implementation.

Set Flatbuffer verification parameters to the same settings as the
C++ arrow implementation (ARROW-11559). This change allows reading
IPC data with more than 1 milion columns.

Closes: jorgecarleitao#231
@ghuls ghuls mentioned this pull request Jul 30, 2021
Closed
@jorgecarleitao jorgecarleitao changed the title Set Flatbuffer verification parameters to the same settings as the C+… Added support for reading IPC with more than 1 million columns Jul 30, 2021
@codecov
Copy link

codecov bot commented Jul 30, 2021

Codecov Report

Merging #240 (771fd2a) into main (526fd1d) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #240      +/-   ##
==========================================
- Coverage   76.82%   76.81%   -0.02%     
==========================================
  Files         229      229              
  Lines       19589    19617      +28     
==========================================
+ Hits        15049    15068      +19     
- Misses       4540     4549       +9
Impacted Files Coverage Δ
src/io/ipc/read/reader.rs 86.66% <100.00%> (+0.11%) ⬆️
src/array/growable/boolean.rs 84.84% <0.00%> (-0.87%) ⬇️
src/array/growable/binary.rs 97.29% <0.00%> (-0.08%) ⬇️
src/array/growable/list.rs 98.36% <0.00%> (-0.04%) ⬇️
src/compute/filter.rs 63.19% <0.00%> (ø)
src/array/growable/primitive.rs 100.00% <0.00%> (ø)
src/compute/merge_sort/mod.rs 94.09% <0.00%> (+0.53%) ⬆️
src/array/growable/mod.rs 36.23% <0.00%> (+1.53%) ⬆️
src/array/growable/fixed_binary.rs 86.48% <0.00%> (+2.73%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 526fd1d...771fd2a. Read the comment docs.

@jorgecarleitao jorgecarleitao added enhancement An improvement to an existing feature and removed enhancement An improvement to an existing feature labels Jul 30, 2021
jorgecarleitao
jorgecarleitao approved these changes Jul 30, 2021
View reviewed changes
Copy link
Owner

@jorgecarleitao jorgecarleitao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome; thanks!

@jorgecarleitao jorgecarleitao merged commit 8e146a7 into jorgecarleitao:main Jul 31, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@jorgecarleitao jorgecarleitao jorgecarleitao approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support loading Feather v2 (IPC) files with more than 1 million tables
2 participants
@ghuls @jorgecarleitao