Parquet Modular decryption support #6637
Conversation
|
Currently this is a rough rebase of work done by @ggershinsky. As |
@rok let me know if you want any help shoehorning this into |
|
Is there any help, input or contribution needed here? |
|
Thanks for the offer @etseidl & @brainslush! I'm making some progress and would definitely appreciate a review! I'll ping once I push. |
rok
force-pushed
the
decryption-basics-fork
branch
2 times, most recently
from
fe488b3 to
d263510
Compare
@etseidl could you please do a quick pass to say if this makes sense in respect to |
parquet/src/file/footer.rs
Outdated
| /// by the [Parquet Spec]. | ||
| /// | ||
| /// [Parquet Spec]: https://github.com/apache/parquet-format#metadata | ||
| #[deprecated(since = "53.1.0", note = "Use ParquetMetaDataReader::decode_metadata")] | ||
| pub fn decode_metadata(buf: &[u8]) -> Result<ParquetMetaData> { | ||
| ParquetMetaDataReader::decode_metadata(buf) | ||
| pub fn decode_metadata( |
There was a problem hiding this comment.
I'm not sure we should be updating a deprecated function. If encryption is desired I'd say force use of the new API so we don't have to maintain this one. Just pass None to ParquetMetaDataReader::decode_metadata.
parquet/src/file/metadata/reader.rs
Outdated
| &mut fetch, | ||
| file_size, | ||
| self.get_prefetch_size(), | ||
| self.file_decryption_properties.clone(), |
There was a problem hiding this comment.
Very minor nit: I understand that file_decryption_properties needs to be cloned eventually...just wondering if we could pass references down into decode_metadata and do the clone there where it's more obviously needed.
rok
force-pushed
the
decryption-basics-fork
branch
from
f90d8b4 to
29d55eb
Compare
adamreeve
force-pushed
the
decryption-basics-fork
branch
from
27e77ad to
7db06cc
Compare
rok
force-pushed
the
decryption-basics-fork
branch
from
a4105d5 to
3e7646d
Compare
rok
force-pushed
the
decryption-basics-fork
branch
2 times, most recently
from
7ac53ba to
deedba9
Compare
rok
force-pushed
the
decryption-basics-fork
branch
from
deedba9 to
951f2fa
Compare
rok
changed the title
rok
force-pushed
the
decryption-basics-fork
branch
from
6f3f0be to
6acb984
Compare
rok
force-pushed
the
decryption-basics-fork
branch
from
f6b9e88 to
23375d1
Compare
adamreeve
force-pushed
the
decryption-basics-fork
branch
2 times, most recently
from
ccdac56 to
7f94e39
Compare
adamreeve
force-pushed
the
decryption-basics-fork
branch
from
7f94e39 to
177d826
Compare
Which issue does this PR close?
This PR is based on branch and an internal patch and aims to provide basic modular decryption support. Partially closes #3511. We decided to split encryption work into a separate PR.
Rationale for this change
See #3511.
What changes are included in this PR?
This introduces AesGcmV1 cypher decryption to
ArrowReaderMetadataandParquetRecordBatchReader. Introduced classes and functions are tested on sample files fromparquet-dataset.Are there any user-facing changes?
Several new classes and method parameters are introduced. If project is compiled without
encryptionflag changes are not breaking. Ifencryptionflag is on some methods and constructors (e.g.ParquetMetaData::new) will require new parameters which would be a breaking change.