Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lz4Raw compression failed if input is tiny #117

Closed
dantengsky opened this issue Apr 6, 2022 · 2 comments
Closed

Lz4Raw compression failed if input is tiny #117

dantengsky opened this issue Apr 6, 2022 · 2 comments

Comments

@dantengsky
Copy link
Contributor

When writing parquet file with the compression option Compression::Lz4Raw, if the input buffer is tiny, the writer fails with error

Error: General("underlying IO error: Compression failed")

e.g.

if we change the following lines

"id" => {
let expected = vec![4, 5, 6, 7, 2, 3, 0, 1];
let expected = expected.into_iter().map(Some).collect::<Vec<_>>();
Array::Int32(expected)

to

        "id" => {
            let expected = vec![4];
            let expected = expected.into_iter().map(Some).collect::<Vec<_>>();
            Array::Int32(expected)

cargo test int32_lz4 will output

---- write::int32_lz4 stdout ----
Error: General("underlying IO error: Compression failed")
.....
@dantengsky
Copy link
Contributor Author

Sorry, PR opened in the incorrect repo :(

@jorgecarleitao
Copy link
Owner

Closed by #118

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants