Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Implement full RleDictionary decoding #400

Closed
Dandandan opened this issue Sep 13, 2021 · 2 comments · Fixed by #402
Closed

Implement full RleDictionary decoding #400

Dandandan opened this issue Sep 13, 2021 · 2 comments · Fixed by #402
Labels
enhancement An improvement to an existing feature good first issue Good for newcomers no-changelog Issues whose changes are covered by a PR and thus should not be shown in the changelog

Comments

@Dandandan
Copy link
Collaborator

When running against a parquet file generated from parquet (via datafusion tpch benchmark conversion tool)
I got the following error:

Arrow error: Not yet implemented: Decoding \"RleDictionary\"-encoded, dictionary-encoded required V1 pages is not yet implemented for primitive"
@jorgecarleitao jorgecarleitao added enhancement An improvement to an existing feature good first issue Good for newcomers labels Sep 13, 2021
@jorgecarleitao
Copy link
Owner

If someone would like to approach this, a design is to write a read_dict_buffer_required based on read_dict_buffer_optional that has no "definition levels", and thus we only need to extend the MutableBuffer with the values and MutableBitmap with a constant true, and then call the function on a new branch of the extend_from_page branch for

(Encoding::PlainDictionary | Encoding::RleDictionary, Some(dict), false)

@Dandandan
Copy link
Collaborator Author

I am giving a shot at this

@jorgecarleitao jorgecarleitao added no-changelog Issues whose changes are covered by a PR and thus should not be shown in the changelog good first issue Good for newcomers and removed good first issue Good for newcomers labels Sep 13, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement An improvement to an existing feature good first issue Good for newcomers no-changelog Issues whose changes are covered by a PR and thus should not be shown in the changelog
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants