Print Parquet BasicTypeInfo id when present #7094

devinrsmith · 2025-02-07T05:53:14Z

Closes [parquet] Print Parquet BasicTypeInfo id when present #7081

alamb

Thanks @devinrsmith -- looks good to me

Any chance you can also improve documentation of print_schema with an example of the output too? 🙏 (not needed to merge this PR, just a request)

So for example, update https://arrow.apache.org/rust/parquet/schema/printer/fn.print_schema.html with the output of your test:

message schema [2] {
 OPTIONAL group field [1] {
    REQUIRED INT32 f1 [0] (INT_32);
    OPTIONAL BYTE_ARRAY f2 [1] (UTF8);
    OPTIONAL BYTE_ARRAY f3 [1] (STRING);
  }
  REPEATED FIXED_LEN_BYTE_ARRAY (12) f4 [2] (INTERVAL);
}";

alamb · 2025-02-07T13:01:43Z

parquet/src/schema/printer.rs

+                .build();
+            let f3 = Type::primitive_type_builder("f3", PhysicalType::BYTE_ARRAY)
+                .with_logical_type(Some(LogicalType::String))
+                .with_id(Some(1))


is it a problem that this is the same id as f2?

Parquet itself treats the ids as metadata, it does not care about them. Higher level systems that map their own schemas into parquet schemas (such as Iceberg) may have stricter requirements and care.

alamb · 2025-02-07T13:01:48Z

parquet/src/schema/printer.rs

+                .unwrap();
+            p.print(&message);
+        }
+        let expected = "message schema [2] {


I verified that without the changes in this PR the output looks like the following (no ids) 👍

devinrsmith · 2025-02-07T15:40:55Z

Thanks @alamb for the quick review!

Any chance you can also improve documentation of print_schema with an example of the output too?

I can add that into this PR, no problem; I'll have that out to you late today.

alamb · 2025-02-08T13:23:30Z

parquet/src/schema/printer.rs

+///
+/// outputs
+///
+/// ```text


alamb · 2025-02-08T14:34:43Z

Thanks again @devinrsmith ❤️

* Print Parquet BasicTypeInfo id when present * Improve print_schema documentation * tiny cleanup

…e32 (#7074) * Support converting large dates (i.e. +10999-12-31) from string to Date32 * Fix lint * Update arrow-cast/src/parse.rs Co-authored-by: Andrew Lamb <[email protected]> * fix: issue introduced in #6833 - less than equal check for scale in decimal conversion (#7070) * fix <= check for scale in decimal conversion * Update arrow-cast/src/cast/mod.rs name change Co-authored-by: Arttu <[email protected]> * remove incorrect comment --------- Co-authored-by: Arttu <[email protected]> * minor: re-export `OffsetBufferBuilder` in `arrow` crate (#7077) * Add another decimal cast edge test case (#7078) * Add another decimal cast edge test case Before 1019f5b this test would fail, as the cast produced 1. 0 is an edge case worth explicitly testing for. * typo/fmt Co-authored-by: Felipe Oliveira Carvalho <[email protected]> --------- Co-authored-by: Felipe Oliveira Carvalho <[email protected]> * Support both 0x01 and 0x02 as type for list of booleans in thrift metadata (#7052) * Support both 0x01 and 0x02 as type for list of booleans * Also support 0 for false inside boolean collections * Use hex notation in tests * Fix LocalFileSystem with range request that ends beyond end of file (#6751) * Fix LocalFileSystem with range request that ends beyond end of file * fix windows * add comment * Seek error * fix seek check * remove windows flag * Get file length from file metadata * Introduce `UnsafeFlag` to manage disabling `ArrayData` validation (#7027) * Introduce UnsafeFlag to manage disabling validation * fix docs * Refactor arrow-ipc: Rename `ArrayReader` to `RecodeBatchDecoder` (#7028) * Rename `ArrayReader` to `RecordBatchDecoder` * Remove alias for `self` * Minor: Update release schedule (#7086) * Minor: Update release schedule * realism * Refactor some decimal-related code and tests (#7062) * Refactor some decimal-related code and tests in preparation for adding Decimal32 and Decimal64 support * Fixed symbol * Apply PR feedback * Fixed format problem * Fixed logical merge conflicts * PR feedback * Refactor arrow-ipc: Move `create_*_array` methods into `RecordBatchDecoder` (#7029) * Move `create_primitive_array` into RecordBatchReader * Move `create_list-array` into RecordBatchReader * Move `create_dictionay_array` into RecordBatchReader * Print Parquet BasicTypeInfo id when present (#7094) * Print Parquet BasicTypeInfo id when present * Improve print_schema documentation * tiny cleanup * Add a custom implementation `LocalFileSystem::list_with_offset` (#7019) * Initial change from Daniel. * Upgrade unit test to be more generic. * Add comments on why we have filter * Cleanup unit tests. * Update object_store/src/local.rs Co-authored-by: Adam Reeve <[email protected]> * Add changes suggested by Adam. * Cleanup match error. * Apply formatting changes suggested by cargo +stable fmt --all. * Apply cosmetic changes suggested by clippy. * Upgrade test_path_with_offset to create temporary directory + files for testing rather than pointing to existing dir. --------- Co-authored-by: Adam Reeve <[email protected]> * fix: first none/empty list in `ListArray` panics in `cast_with_options` (#7065) * fix: first none in `ListArray` panics in `cast_with_options` * simplify * fix * Update arrow-cast/src/cast/list.rs Co-authored-by: Jeffrey Vo <[email protected]> --------- Co-authored-by: Jeffrey Vo <[email protected]> * Benchmarks for Arrow IPC writer (#7090) * Add benchmarks for Arrow IPC writer * Add benchmarks for Arrow IPC writer * reuse target buffer * rename, etc * Add compression type * update --------- Co-authored-by: Andy Grove <[email protected]> * Minor: Clarify documentation on `NullBufferBuilder::allocated_size` (#7089) * Minor: Clarify documentaiton on NullBufferBuilder::allocated_size * add note about why allocations are 64 bytes * Add more tests for edge cases * Add negative test case for incorrectly formatted large dates --------- Co-authored-by: Andrew Lamb <[email protected]> Co-authored-by: Himadri Pal <[email protected]> Co-authored-by: Arttu <[email protected]> Co-authored-by: Piotr Findeisen <[email protected]> Co-authored-by: Felipe Oliveira Carvalho <[email protected]> Co-authored-by: Jörn Horstmann <[email protected]> Co-authored-by: Kyle Barron <[email protected]> Co-authored-by: Curt Hagenlocher <[email protected]> Co-authored-by: Devin Smith <[email protected]> Co-authored-by: Corwin Joy <[email protected]> Co-authored-by: Adam Reeve <[email protected]> Co-authored-by: irenjj <[email protected]> Co-authored-by: Jeffrey Vo <[email protected]> Co-authored-by: Andy Grove <[email protected]>

Print Parquet BasicTypeInfo id when present

2c2b99b

github-actions bot added the parquet Changes to the parquet crate label Feb 7, 2025

alamb approved these changes Feb 7, 2025

View reviewed changes

Improve print_schema documentation

b39cfcb

devinrsmith requested a review from alamb February 8, 2025 06:13

tiny cleanup

09d890b

alamb reviewed Feb 8, 2025

View reviewed changes

parquet/src/schema/printer.rs

///

/// outputs

///

/// ```text

Copy link

Contributor

alamb Feb 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

alamb merged commit 4e34203 into apache:main Feb 8, 2025
16 checks passed

phillipleblanc pushed a commit to spiceai/arrow-rs that referenced this pull request Feb 10, 2025

Print Parquet BasicTypeInfo id when present (apache#7094)

0d943b9

* Print Parquet BasicTypeInfo id when present * Improve print_schema documentation * tiny cleanup

alamb mentioned this pull request Feb 10, 2025

[parquet] Print Parquet BasicTypeInfo id when present #7081

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Print Parquet BasicTypeInfo id when present #7094

Print Parquet BasicTypeInfo id when present #7094

devinrsmith commented Feb 7, 2025 •

edited by alamb

Loading

alamb left a comment

alamb Feb 7, 2025

devinrsmith Feb 7, 2025

alamb Feb 7, 2025

devinrsmith commented Feb 7, 2025

alamb Feb 8, 2025

alamb commented Feb 8, 2025

Print Parquet BasicTypeInfo id when present #7094

Print Parquet BasicTypeInfo id when present #7094

Conversation

devinrsmith commented Feb 7, 2025 • edited by alamb Loading

alamb left a comment

Choose a reason for hiding this comment

alamb Feb 7, 2025

Choose a reason for hiding this comment

devinrsmith Feb 7, 2025

Choose a reason for hiding this comment

alamb Feb 7, 2025

Choose a reason for hiding this comment

devinrsmith commented Feb 7, 2025

alamb Feb 8, 2025

Choose a reason for hiding this comment

alamb commented Feb 8, 2025

devinrsmith commented Feb 7, 2025 •

edited by alamb

Loading