Minor: Clarify documentation on `NullBufferBuilder::allocated_size` #7089

alamb · 2025-02-06T19:18:17Z

Which issue does this PR close?

Related to

refactor: switch BooleanBufferBuilder to NullBufferBuilder in MaybeNullBufferBuilder datafusion#14504

Rationale for this change

@Chen-Yuan-Lai found in apache/datafusion#14504 that NullBufferBuilder::allocated_size actually reports the size in bits, not bytes -- nice eyes!

What changes are included in this PR?

Update the documentation to reflect what the code does

Add an example onBooleanBufferBuilder::capacity to verify it is in terms of bits

Are there any user-facing changes?

Better docs

alamb · 2025-02-06T19:18:50Z

arrow-buffer/src/builder/boolean.rs

@@ -83,7 +83,23 @@ impl BooleanBufferBuilder {
        self.len == 0
    }

-    /// Returns the capacity of the buffer
+    /// Returns the capacity of the buffer, in bits (not bytes)


The key detail is that this function returns the value in bits (not bytes)

alamb · 2025-02-06T19:19:34Z

arrow-buffer/src/builder/null.rs

@@ -217,7 +217,7 @@ impl NullBufferBuilder {
        self.bitmap_builder.as_mut().map(|b| b.as_slice_mut())
    }

-    /// Return the allocated size of this builder, in bytes, useful for memory accounting.
+    /// Return the allocated size of this builder, in bits, useful for memory accounting.


This documentation is wrong -- as it uses capacity() directly which returns a size in bits

I actually think it would be better if we had a function that returned allocated size in bytes (maybe we can deprecate this one and make a new one like allocated_bytes() or something for clarity.

IMO this is a bug and we could fix this without it being a breaking change

I created #7121

Jefffrey

This is something I also noticed in #7082

One thing to note is that NullBufferBuilder::allocated_size() is used here:

arrow-rs/arrow-array/src/builder/generic_bytes_view_builder.rs

Lines 397 to 408 in 92cfd99

    
               /// Return the allocated size of this builder in bytes, useful for memory accounting. 
        
               pub fn allocated_size(&self) -> usize { 
        
                   let views = self.views_builder.capacity() * std::mem::size_of::<u128>(); 
        
                   let null = self.null_buffer_builder.allocated_size(); 
        
                   let buffer_size = self.completed.iter().map(|b| b.capacity()).sum::<usize>(); 
        
                   let in_progress = self.in_progress.capacity(); 
        
                   let tracker = match &self.string_tracker { 
        
                       Some((ht, _)) => ht.capacity() * std::mem::size_of::<usize>(), 
        
                       None => 0, 
        
                   }; 
        
                   buffer_size + in_progress + tracker + views + null 
        
               }

And I think it's used with assumption it provides bytes not bits, so may need adjustment.

Jefffrey · 2025-02-07T09:26:54Z

arrow-buffer/src/builder/boolean.rs

+    /// // empty requires 0 bytes
+    /// let b = BooleanBufferBuilder::new(0);
+    /// assert_eq!(0, b.capacity());
+    /// // Creating space for 1 bit results in 64 bytes (space for 512 bits)


Maybe add a small note here explaining why it over allocates?

Done in 12055a0

alamb · 2025-02-08T16:11:55Z

This is something I also noticed in #7082

Filed Memory accounting in GenericBytesView overcounts allocation size #7099

Jefffrey · 2025-02-09T01:14:15Z

👍

…pache#7089) * Minor: Clarify documentaiton on NullBufferBuilder::allocated_size * add note about why allocations are 64 bytes

alamb · 2025-02-12T01:07:14Z

Thanks @tustvold and @Jefffrey

…e32 (#7074) * Support converting large dates (i.e. +10999-12-31) from string to Date32 * Fix lint * Update arrow-cast/src/parse.rs Co-authored-by: Andrew Lamb <[email protected]> * fix: issue introduced in #6833 - less than equal check for scale in decimal conversion (#7070) * fix <= check for scale in decimal conversion * Update arrow-cast/src/cast/mod.rs name change Co-authored-by: Arttu <[email protected]> * remove incorrect comment --------- Co-authored-by: Arttu <[email protected]> * minor: re-export `OffsetBufferBuilder` in `arrow` crate (#7077) * Add another decimal cast edge test case (#7078) * Add another decimal cast edge test case Before 1019f5b this test would fail, as the cast produced 1. 0 is an edge case worth explicitly testing for. * typo/fmt Co-authored-by: Felipe Oliveira Carvalho <[email protected]> --------- Co-authored-by: Felipe Oliveira Carvalho <[email protected]> * Support both 0x01 and 0x02 as type for list of booleans in thrift metadata (#7052) * Support both 0x01 and 0x02 as type for list of booleans * Also support 0 for false inside boolean collections * Use hex notation in tests * Fix LocalFileSystem with range request that ends beyond end of file (#6751) * Fix LocalFileSystem with range request that ends beyond end of file * fix windows * add comment * Seek error * fix seek check * remove windows flag * Get file length from file metadata * Introduce `UnsafeFlag` to manage disabling `ArrayData` validation (#7027) * Introduce UnsafeFlag to manage disabling validation * fix docs * Refactor arrow-ipc: Rename `ArrayReader` to `RecodeBatchDecoder` (#7028) * Rename `ArrayReader` to `RecordBatchDecoder` * Remove alias for `self` * Minor: Update release schedule (#7086) * Minor: Update release schedule * realism * Refactor some decimal-related code and tests (#7062) * Refactor some decimal-related code and tests in preparation for adding Decimal32 and Decimal64 support * Fixed symbol * Apply PR feedback * Fixed format problem * Fixed logical merge conflicts * PR feedback * Refactor arrow-ipc: Move `create_*_array` methods into `RecordBatchDecoder` (#7029) * Move `create_primitive_array` into RecordBatchReader * Move `create_list-array` into RecordBatchReader * Move `create_dictionay_array` into RecordBatchReader * Print Parquet BasicTypeInfo id when present (#7094) * Print Parquet BasicTypeInfo id when present * Improve print_schema documentation * tiny cleanup * Add a custom implementation `LocalFileSystem::list_with_offset` (#7019) * Initial change from Daniel. * Upgrade unit test to be more generic. * Add comments on why we have filter * Cleanup unit tests. * Update object_store/src/local.rs Co-authored-by: Adam Reeve <[email protected]> * Add changes suggested by Adam. * Cleanup match error. * Apply formatting changes suggested by cargo +stable fmt --all. * Apply cosmetic changes suggested by clippy. * Upgrade test_path_with_offset to create temporary directory + files for testing rather than pointing to existing dir. --------- Co-authored-by: Adam Reeve <[email protected]> * fix: first none/empty list in `ListArray` panics in `cast_with_options` (#7065) * fix: first none in `ListArray` panics in `cast_with_options` * simplify * fix * Update arrow-cast/src/cast/list.rs Co-authored-by: Jeffrey Vo <[email protected]> --------- Co-authored-by: Jeffrey Vo <[email protected]> * Benchmarks for Arrow IPC writer (#7090) * Add benchmarks for Arrow IPC writer * Add benchmarks for Arrow IPC writer * reuse target buffer * rename, etc * Add compression type * update --------- Co-authored-by: Andy Grove <[email protected]> * Minor: Clarify documentation on `NullBufferBuilder::allocated_size` (#7089) * Minor: Clarify documentaiton on NullBufferBuilder::allocated_size * add note about why allocations are 64 bytes * Add more tests for edge cases * Add negative test case for incorrectly formatted large dates --------- Co-authored-by: Andrew Lamb <[email protected]> Co-authored-by: Himadri Pal <[email protected]> Co-authored-by: Arttu <[email protected]> Co-authored-by: Piotr Findeisen <[email protected]> Co-authored-by: Felipe Oliveira Carvalho <[email protected]> Co-authored-by: Jörn Horstmann <[email protected]> Co-authored-by: Kyle Barron <[email protected]> Co-authored-by: Curt Hagenlocher <[email protected]> Co-authored-by: Devin Smith <[email protected]> Co-authored-by: Corwin Joy <[email protected]> Co-authored-by: Adam Reeve <[email protected]> Co-authored-by: irenjj <[email protected]> Co-authored-by: Jeffrey Vo <[email protected]> Co-authored-by: Andy Grove <[email protected]>

Minor: Clarify documentaiton on NullBufferBuilder::allocated_size

7ea8bda

alamb added the documentation Improvements or additions to documentation label Feb 6, 2025

github-actions bot added the arrow Changes to the arrow crate label Feb 6, 2025

alamb commented Feb 6, 2025

View reviewed changes

alamb mentioned this pull request Feb 6, 2025

refactor: switch BooleanBufferBuilder to NullBufferBuilder in MaybeNullBufferBuilder apache/datafusion#14504

Merged

Jefffrey approved these changes Feb 7, 2025

View reviewed changes

Merge remote-tracking branch 'apache/main'

02d09c5

alamb changed the title ~~Minor: Clarify documentaiton on NullBufferBuilder::allocated_size~~ Minor: Clarify documentation on NullBufferBuilder::allocated_size Feb 8, 2025

add note about why allocations are 64 bytes

12055a0

alamb mentioned this pull request Feb 8, 2025

Memory accounting in GenericBytesView overcounts allocation size #7099

Open

Jefffrey merged commit c245a45 into apache:main Feb 9, 2025
26 checks passed

alamb deleted the main branch February 9, 2025 10:49

This was referenced Feb 11, 2025

bugfix: adjust allocated_size() in GenericByteViewBuilder #7104

Draft

NullBufferBuilder::allocated_size Returns Size in Bits #7121

Closed

tustvold mentioned this pull request Feb 12, 2025

fix: NullBufferBuilder::allocated_size should return Size in Bytes #7122

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Minor: Clarify documentation on `NullBufferBuilder::allocated_size` #7089

Minor: Clarify documentation on `NullBufferBuilder::allocated_size` #7089

alamb commented Feb 6, 2025

alamb Feb 6, 2025

alamb Feb 6, 2025 •

edited

Loading

tustvold Feb 11, 2025

tustvold Feb 11, 2025

Jefffrey left a comment

Jefffrey Feb 7, 2025

alamb Feb 8, 2025

alamb commented Feb 8, 2025

Jefffrey commented Feb 9, 2025

alamb commented Feb 12, 2025

	/// Return the allocated size of this builder in bytes, useful for memory accounting.
	pub fn allocated_size(&self) -> usize {
	let views = self.views_builder.capacity() * std::mem::size_of::<u128>();
	let null = self.null_buffer_builder.allocated_size();
	let buffer_size = self.completed.iter().map(\|b\| b.capacity()).sum::<usize>();
	let in_progress = self.in_progress.capacity();
	let tracker = match &self.string_tracker {
	Some((ht, _)) => ht.capacity() * std::mem::size_of::<usize>(),
	None => 0,
	};
	buffer_size + in_progress + tracker + views + null
	}

Minor: Clarify documentation on NullBufferBuilder::allocated_size #7089

Minor: Clarify documentation on NullBufferBuilder::allocated_size #7089

Conversation

alamb commented Feb 6, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

alamb Feb 6, 2025

Choose a reason for hiding this comment

alamb Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

tustvold Feb 11, 2025

Choose a reason for hiding this comment

tustvold Feb 11, 2025

Choose a reason for hiding this comment

Jefffrey left a comment

Choose a reason for hiding this comment

Jefffrey Feb 7, 2025

Choose a reason for hiding this comment

alamb Feb 8, 2025

Choose a reason for hiding this comment

alamb commented Feb 8, 2025

Jefffrey commented Feb 9, 2025

alamb commented Feb 12, 2025

Minor: Clarify documentation on `NullBufferBuilder::allocated_size` #7089

Minor: Clarify documentation on `NullBufferBuilder::allocated_size` #7089

alamb Feb 6, 2025 •

edited

Loading