Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Reading compressed file produces an error: failed to fill whole buffer #590

Closed
dimasukr opened this issue Nov 9, 2021 · 0 comments · Fixed by #593
Closed

Reading compressed file produces an error: failed to fill whole buffer #590

dimasukr opened this issue Nov 9, 2021 · 0 comments · Fixed by #593
Labels
bug Something isn't working no-changelog Issues whose changes are covered by a PR and thus should not be shown in the changelog

Comments

@dimasukr
Copy link

dimasukr commented Nov 9, 2021

I'm trying new feature added in #566
and I'm getting an error from Reader: failed to fill whole buffer

Here is my sample code:

use arrow2::array::Int32Array;
use arrow2::datatypes::{DataType, Field, Schema};
use arrow2::error::Result;
use arrow2::io::ipc::read::{read_file_metadata, FileReader};
use arrow2::io::ipc::write;
use arrow2::io::ipc::write::Compression;
use arrow2::record_batch::RecordBatch;
use std::fs::File;
use std::sync::Arc;

fn main() -> Result<()> {
    let schema = Schema::new(vec![Field::new("a", DataType::Int32, false)]);

    let a = Int32Array::from_slice(&[0; 10]);
    let batch = RecordBatch::try_new(Arc::new(schema.clone()), vec![Arc::new(a)])?;

    let file = File::create("test.arrow")?;
    let options = write::WriteOptions {
        compression: Some(Compression::LZ4),
    };
    let mut writer = write::FileWriter::try_new(file, &schema, options)?;
    writer.write(&batch)?;
    writer.finish()?;

    let mut file = File::open("test.arrow")?;
    let metadata = read_file_metadata(&mut file)?;
    let reader = FileReader::new(file, metadata, None);
    reader.collect::<Result<Vec<RecordBatch>>>()?;
    Ok(())
}

It produces:

Error: Io(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })

I'm running it on latest nightly but I don't think it makes any difference.

@jorgecarleitao jorgecarleitao added bug Something isn't working no-changelog Issues whose changes are covered by a PR and thus should not be shown in the changelog labels Nov 10, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working no-changelog Issues whose changes are covered by a PR and thus should not be shown in the changelog
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants