Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix LocalFileSystem with range request that ends beyond end of file #6751

Merged
merged 8 commits into from
Feb 6, 2025

Conversation

kylebarron
Copy link
Contributor

Which issue does this PR close?

Closes #6749.

Rationale for this change

Make LocalFileSystem act the same as other object store backends.

What changes are included in this PR?

Allow LocalFileSystem to return a satisfiable byte range beyond the end of the file.

Add new test for satisfiable and non-satisfiable byte range.

Are there any user-facing changes?

No.

@github-actions github-actions bot added the object-store Object Store Interface label Nov 18, 2024
Copy link
Contributor

@etseidl etseidl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like several tests are failing because neither the seek nor the read return an error when seeking/reading beyond the end of the file. The Seek and Read traits seem to allow operations beyond EOF...read in particular will simply return 0 rather than produce an error. Perhaps read_range should not test the number of bytes returned at all and return what it can, leaving it up to the caller to detect when a read returns fewer bytes than requested and act accordingly. 🤷

Edit:
Ok, I read the discord discussion, so ignore the above. Instead, how about getting the actual file length from the File metadata, and then test that range.start is not beyond that?

    let file_len = metadata.len();
    ensure!(
        file_len > range.start as u64,
        OutOfRangeSnafu {
            path,
            expected: range.start,
            actual: file_len as usize
        }
    );

    let to_read = range.end - range.start;
    let seek_idx = file
        .seek(SeekFrom::Start(range.start as u64))
        .context(SeekSnafu { path })?;

object_store/src/local.rs Outdated Show resolved Hide resolved
object_store/src/local.rs Outdated Show resolved Hide resolved
@alamb
Copy link
Contributor

alamb commented Nov 22, 2024

🤔 the CI failures appear to be real

@tustvold tustvold marked this pull request as draft December 12, 2024 21:20
@kylebarron kylebarron marked this pull request as ready for review February 5, 2025 16:10
@kylebarron
Copy link
Contributor Author

I updated this to use File::metadata to access the length of the file and tests now pass locally for me.

@kylebarron kylebarron requested a review from etseidl February 5, 2025 16:17
Copy link
Contributor

@etseidl etseidl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, and is now passing tests.

@alamb
Copy link
Contributor

alamb commented Feb 6, 2025

❤️ --- thanks @kylebarron and @etseidl for the review!

@alamb alamb merged commit 7a15e4b into apache:main Feb 6, 2025
14 checks passed
phillipleblanc pushed a commit to spiceai/arrow-rs that referenced this pull request Feb 10, 2025
…pache#6751)

* Fix LocalFileSystem with range request that ends beyond end of file

* fix windows

* add comment

* Seek error

* fix seek check

* remove windows flag

* Get file length from file metadata
alamb added a commit that referenced this pull request Feb 12, 2025
…e32 (#7074)

* Support converting large dates (i.e. +10999-12-31) from string to Date32

* Fix lint

* Update arrow-cast/src/parse.rs

Co-authored-by: Andrew Lamb <[email protected]>

* fix: issue introduced in #6833 -  less than equal check for scale in decimal conversion (#7070)

* fix <= check for scale in decimal conversion

* Update arrow-cast/src/cast/mod.rs

name change

Co-authored-by: Arttu <[email protected]>

* remove incorrect comment

---------

Co-authored-by: Arttu <[email protected]>

* minor: re-export `OffsetBufferBuilder` in `arrow` crate (#7077)

* Add another decimal cast edge test case (#7078)

* Add another decimal cast edge test case

Before 1019f5b this test would fail, as
the cast produced 1. 0 is an edge case worth explicitly testing for.

* typo/fmt

Co-authored-by: Felipe Oliveira Carvalho <[email protected]>

---------

Co-authored-by: Felipe Oliveira Carvalho <[email protected]>

* Support both 0x01 and 0x02 as type for list of booleans in thrift metadata (#7052)

* Support both 0x01 and 0x02 as type for list of booleans

* Also support 0 for false inside boolean collections

* Use hex notation in tests

* Fix LocalFileSystem with range request that ends beyond end of file (#6751)

* Fix LocalFileSystem with range request that ends beyond end of file

* fix windows

* add comment

* Seek error

* fix seek check

* remove windows flag

* Get file length from file metadata

* Introduce `UnsafeFlag` to manage disabling `ArrayData` validation (#7027)

* Introduce UnsafeFlag to manage disabling validation

* fix docs

* Refactor arrow-ipc: Rename `ArrayReader` to `RecodeBatchDecoder` (#7028)

* Rename `ArrayReader` to `RecordBatchDecoder`

* Remove alias for `self`

* Minor: Update release schedule (#7086)

* Minor: Update release schedule

* realism

* Refactor some decimal-related code and tests (#7062)

* Refactor some decimal-related code and tests in preparation for adding Decimal32 and Decimal64 support

* Fixed symbol

* Apply PR feedback

* Fixed format problem

* Fixed logical merge conflicts

* PR feedback

* Refactor arrow-ipc: Move `create_*_array` methods into `RecordBatchDecoder` (#7029)

* Move `create_primitive_array` into RecordBatchReader

* Move `create_list-array` into RecordBatchReader

* Move `create_dictionay_array` into RecordBatchReader

* Print Parquet BasicTypeInfo id when present (#7094)

* Print Parquet BasicTypeInfo id when present

* Improve print_schema documentation

* tiny cleanup

* Add a custom implementation `LocalFileSystem::list_with_offset`  (#7019)

* Initial change from Daniel.

* Upgrade unit test to be more generic.

* Add comments on why we have filter

* Cleanup unit tests.

* Update object_store/src/local.rs

Co-authored-by: Adam Reeve <[email protected]>

* Add changes suggested by Adam.

* Cleanup match error.

* Apply formatting changes suggested by cargo +stable fmt --all.

* Apply cosmetic changes suggested by clippy.

* Upgrade test_path_with_offset to create temporary directory + files for testing rather than pointing to existing dir.

---------

Co-authored-by: Adam Reeve <[email protected]>

* fix: first none/empty list in `ListArray` panics in `cast_with_options` (#7065)

* fix: first none in `ListArray` panics in `cast_with_options`

* simplify

* fix

* Update arrow-cast/src/cast/list.rs

Co-authored-by: Jeffrey Vo <[email protected]>

---------

Co-authored-by: Jeffrey Vo <[email protected]>

* Benchmarks for Arrow IPC writer (#7090)

* Add benchmarks for Arrow IPC writer

* Add benchmarks for Arrow IPC writer

* reuse target buffer

* rename, etc

* Add compression type

* update

---------

Co-authored-by: Andy Grove <[email protected]>

* Minor: Clarify documentation on `NullBufferBuilder::allocated_size` (#7089)

* Minor: Clarify documentaiton on NullBufferBuilder::allocated_size

* add note about why allocations are 64 bytes

* Add more tests for edge cases

* Add negative test case for incorrectly formatted large dates

---------

Co-authored-by: Andrew Lamb <[email protected]>
Co-authored-by: Himadri Pal <[email protected]>
Co-authored-by: Arttu <[email protected]>
Co-authored-by: Piotr Findeisen <[email protected]>
Co-authored-by: Felipe Oliveira Carvalho <[email protected]>
Co-authored-by: Jörn Horstmann <[email protected]>
Co-authored-by: Kyle Barron <[email protected]>
Co-authored-by: Curt Hagenlocher <[email protected]>
Co-authored-by: Devin Smith <[email protected]>
Co-authored-by: Corwin Joy <[email protected]>
Co-authored-by: Adam Reeve <[email protected]>
Co-authored-by: irenjj <[email protected]>
Co-authored-by: Jeffrey Vo <[email protected]>
Co-authored-by: Andy Grove <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
object-store Object Store Interface
Projects
None yet
Development

Successfully merging this pull request may close these issues.

LocalFileSystem errors with satisfiable range request
3 participants