Column with empty dictionary but contains values causes panic

[TurnOfACard]

I found an issue where a small dictionary offset (length of 1) in arrow2::io::parquet::read::deserialize::binary::basic::State::OptionalDictionary causes the parquet reader to panic.

I imagine restricting the slice bound is a fix. In general, there doesn't appear to be a way to propagate errors without some API changes:

• the function the code is located in (extend_from_state) does not return a Result
• either does its caller: extend_from_new_page

I assume that we should modify the return values of extend_from_state and extend_from_new_page?

arrow2/src/io/parquet/read/deserialize/binary/basic.rs

Lines 331 to 340 in 9a38663

	State::OptionalDictionary(page_validity, page_values) => {
	let dict_values = page_values.dict.values();
	let dict_offsets = page_values.dict.offsets();
	let op = move |index: u32| {
	let index = index as usize;
	let dict_offset_i = dict_offsets[index] as usize;
	let dict_offset_ip1 = dict_offsets[index + 1] as usize;
	&dict_values[dict_offset_i..dict_offset_ip1]
	};

[jorgecarleitao]

Thanks for the report!

The invariant is that dict_offsets.len() equals page.num_values() + 1, from here. So, for offset.len() == 1, the dict page is empty? In that case the values page should also also be empty?

Do you have the parquet file available?

[TurnOfACard]

Ah, it seems that helps explain part of it.

Long story short, I was using a file produced by arrow2. In a single column, the first PageType was PageType::DictionaryPage, and the second was PageType::DataPageV2. For the DictionaryPage, the compressed_page_size was == 9, but uncompressed_page_size was == 0 (which may be another problem). Encoding was RleDictionary.

As a result of this, parquet2::page::page_dict::deserialize was passed an empty buf with num_values == 0 (plausible if the uncompressed_page_size is zero) it will return (vec![], vec![0]).

This is ultimately passed to the code below, which will always try to slice dict_values which potentially may be zero-length.

arrow2/src/io/parquet/read/deserialize/binary/basic.rs

Lines 331 to 340 in 9a38663

	State::OptionalDictionary(page_validity, page_values) => {
	let dict_values = page_values.dict.values();
	let dict_offsets = page_values.dict.offsets();
	let op = move |index: u32| {
	let index = index as usize;
	let dict_offset_i = dict_offsets[index] as usize;
	let dict_offset_ip1 = dict_offsets[index + 1] as usize;
	&dict_values[dict_offset_i..dict_offset_ip1]
	};

I am not sure why parquet2 exported a page header with a compressed_page_size of 9, and an uncompressed_page_size of 0, I will take a look at the code that produced it.

If I am interpreting this correctly, there is no guard for a malformed parquet file with an empty dictionary but provided values.