Merge branch 'main' into consolidated_metadata

.github/workflows/ci.yml

@@ -2,9 +2,9 @@ name: build
 
 on:
   push:
-    branches: ["main"]
+    branches: ["main", "v[0-9]+.[0-9]+"]
   pull_request:
-    branches: ["main"]
+    branches: ["main", "v[0-9]+.[0-9]+"]
 
 env:
   CARGO_TERM_COLOR: always
@@ -18,7 +18,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        toolchain: ["stable", "1.76", "beta", "nightly"]
+        toolchain: ["stable", "1.77", "beta", "nightly"]
         features: ["--features \"\"", "--all-features", "--no-default-features"]
     runs-on: ubuntu-latest
     steps:

.gitignore

@@ -23,3 +23,6 @@ Cargo.lock
 # Profiling files
 *.profraw
 lcov.info
+
+# cargo-mutants
+mutants.out

CHANGELOG.md

@@ -7,9 +7,133 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+- Add `ArrayShardedReadableExt::retrieve_encoded_inner_chunk`
+- Add `ArrayShardedReadableExt::inner_chunk_byte_range`
+- Add `ArrayShardedExt::is_exclusively_sharded`
+- Add `ArrayShardedReadableExtCache::array_is_exclusively_sharded`
+- Add `Vlen{Array,Bytes,Utf8}Codec`, replacing `VlenV2Codec`
+- Add `ZstdCodecConfigurationNumCodecs`
+  - Adds support for Zarr V2 `zstd` encoded data created with `numcodecs` < 0.13
+- Add support for pcodec `Auto`, `None`, and `TryLookback` delta specs
+
+### Changed
+- **Breaking**: Seal `Array` extension traits: `ArraySharded[Readable]Ext` and `ArrayChunkCacheExt`
+- **Breaking**: Make `{Array,Bytes}PartialDecoderCache` private
+- **Breaking**: Make `Any` a supertrait of partial encoder/decoder traits
+- **Breaking**: Add `ArrayError::UnsupportedMethod`
+- **Breaking**: Rename `DataType::Binary` to `Bytes` for compatibility with `zarr-python`
+- **Breaking**: Make `array::codec::array_to_bytes::bytes::reverse_endianness` private
+- **Breaking**: Make `VlenV2Codec` private
+- Bump `itertools` to 0.14
+
+### Removed
+- Remove support for pcodec `Try{FloatMult,FloatQuant,IntMult}` mode specs
+  - These may be reimplemented when supported by `zarr-python`/`numcodecs`
+
+### Fixed
+- Cleanup unnecessary lifetime constraints in partial decoders
+- Fix `clippy::useless_conversion` lint
+
+## [0.18.3] - 2024-12-30
+
+### Added
+- impl `From<Node>` for `NodePath` ([#112] by [@niklasmueboe])
+- Add `Group::child[_{group,array}]_paths` ([#112] by [@niklasmueboe])
+
+[#112]: https://github.com/LDeakin/zarrs/pull/112
+
+## [0.18.2] - 2024-12-25
+
+### Added
+- functions to get children of Group ([#104] by [@niklasmueboe])
+  - adds `Group::[async_]children`, `Group::[async_]child_groups`, `Group::[async_]child_arrays`
+- Impl `From<Node>` for `NodeMetadata`
+
+### Changed
+- Reduce metadata code duplication in the `Node` module
+
+[#104]: https://github.com/LDeakin/zarrs/pull/104
+
+## [0.18.1] - 2024-12-17
+
+### Changed
+- Bump `zfp-sys` to 0.3.0
+- Bump `bzip2` to 0.5.0
+- Minor readme/ecosystem updates
+
+### Fixed
+- Fix `unsafe_op_in_unsafe_fn` lint
+- Clarify that the `zstd` codec is draft in docs
+- Clarify that the `gdeflate` codec is experimental in docs
+
+## [0.18.0] - 2024-11-23
+
+### Announcements
+- [`zarrs-python`](https://github.com/ilan-gold/zarrs-python) was recently released
+  - It implements a high-performance codec pipeline for the reference [`zarr-python`](https://github.com/zarr-developers/zarr-python) implementation
+  - It is supported by downstream libraries like `dask`
+  - See [zarr_benchmarks](https://github.com/LDeakin/zarr_benchmarks) for benchmarks
+- [The `zarrs` Book](https://book.zarrs.dev) has been created and is under construction
+
+### Release Highlights
+- Experimental partial encoding support (enabling writing shards incrementally)
+- Improve Zarr V2 interoperability and conversion
+
+### Added
+- Add a `makefile` and simplify `BUILD.md`
+- Add chunk-by-chunk update example in `Array` docs
+- Add `array::copy_fill_value_into()`
+- Add experimental partial encoding support (sync only):
+  - Enables writing shards incrementally
+  - With `{CodecOptions,Config}::experimental_partial_encoding` enabled, `Array::store_{array,chunk}_subset` will use partial encoding
+  - This is an experimental feature for now until it has more comprehensively tested and support is added in the async API
+  - Adds `ArrayPartialEncoderTraits`, `BytesPartialEncoderTraits`, `StoragePartialEncoder`, `ArrayPartialEncoderDefault`, `BytesPartialEncoderDefault`
+  - **Breaking**: Add `{ArrayToArray,ArrayToBytes,BytesToBytes}CodecTraits::partial_encoder`
+- Add `with_` methods to `{Array,Group}MetadataOptions`
+- Add `zarr_v2_to_v3` example
+- Add `{Array,Group}::to_v3()`
+- Add `ShardingCodecBuilder::build_arc()`
+- Add `zarrs::version::version_{str,pre}`
+- Add "The zarrs Book" and `zarrs-python` to docs
+
 ### Changed
 - Bump `unsafe_cell_slice` to 0.2.0
 - **Breaking**: Change `output` parameter of `decode_into` codec trait methods to `&UnsafeCellSlice<u8>`
+- **Breaking**: Add `dynamic()` to all `CodecTraits`
+- **Breaking**: Add `options` parameter to `[Async]ArrayPartialDecoderTraits::partial_decode` and remove `partial_decode_opt`
+- Make `array::update_array_bytes()` public
+- **Breaking**: Bump MSRV to 1.77 (21 March, 2024)
+- Bump `zfp-sys` to 0.2.0
+- Display `ArraySubset` as a list of ranges
+- Relax `output_subset` requirements on `ArrayToBytesCodecTraits::decode_into` and `ArrayPartialDecoderTraits::partial_decode_into`
+  - The subset shape and dimensionality no longer has to match, only the number of elements
+- Bump `pco` (pcodec) to 0.4
+- **Breaking**: Change `experimental_codec_names` config hashmap to `HashMap<String, String>` from `HashMap<&'static str, String>`
+- **Breaking**: Add `name` parameter to `vlen_v2` codec constructors
+- Register `vlen-array`, `vlen-bytes`, and `vlen-utf8` codecs
+- Bump `zarrs_metadata` to 0.2.0
+- Bump `zarrs_storage` to 0.3.0
+- Bump `zarrs_filesystem` to 0.2.0
+- Make `zarrs::version::version_{,major,minor,patch}` const
+
+### Removed
+- Remove `async-recursion` dependency
+- **Breaking**: Remove `Default` implementation for `VlenV2Codec`
+
+### Fixed
+- Fix panics that could occur with with empty byte ranges / empty array subsets in `Array`, `ByteRange` and codec methods
+
+## [0.18.0-beta.0] - 2024-11-15
+
+## [0.17.1] - 2024-10-18
+
+### Added
+ - Add `zarrs_icechunk` to ecosystem docs
+
+### Fixed
+ - Fix `data_key` encoding on windows (it contained '//')
+ - Fix `clippy::needless_lifetimes` lint
 
 ## [0.17.0] - 2024-10-02
 
@@ -1076,7 +1200,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Added
  - Initial public release
 
-[unreleased]: https://github.com/LDeakin/zarrs/compare/zarrs-v0.17.0...HEAD
+[unreleased]: https://github.com/LDeakin/zarrs/compare/zarrs-v0.18.3...HEAD
+[0.18.3]: https://github.com/LDeakin/zarrs/releases/tag/zarrs-v0.18.3
+[0.18.2]: https://github.com/LDeakin/zarrs/releases/tag/zarrs-v0.18.2
+[0.18.1]: https://github.com/LDeakin/zarrs/releases/tag/zarrs-v0.18.1
+[0.18.0]: https://github.com/LDeakin/zarrs/releases/tag/zarrs-v0.18.0
+[0.18.0-beta.0]: https://github.com/LDeakin/zarrs/releases/tag/zarrs-v0.18.0-beta.0
+[0.17.1]: https://github.com/LDeakin/zarrs/releases/tag/zarrs-v0.17.1
 [0.17.0]: https://github.com/LDeakin/zarrs/releases/tag/zarrs-v0.17.0
 [0.17.0-beta.3]: https://github.com/LDeakin/zarrs/releases/tag/zarrs-v0.17.0-beta.3
 [0.17.0-beta.2]: https://github.com/LDeakin/zarrs/releases/tag/zarrs-v0.17.0-beta.2
@@ -1127,3 +1257,4 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 [@lorenzocerrone]: https://github.com/lorenzocerrone
 [@dustinlagoy]: https://github.com/dustinlagoy
 [@sk1p]: https://github.com/sk1p
+[@niklasmueboe]: https://github.com/niklasmueboe

CITATION.cff

@@ -1,8 +1,8 @@
 cff-version: 1.2.0
 message: "If you use this software, please cite it as below."
 title: "zarrs"
-version: 0.15.1
-date-released: 2024-07-11
+version: 0.18.3
+date-released: 2024-12-30
 repository-code: "https://github.com/LDeakin/zarrs"
 url: "https://zarrs.dev"
 abstract: "zarrs is a Rust library for the Zarr storage format for multidimensional arrays and metadata."

Cargo.toml

@@ -13,39 +13,52 @@ members = [
     "zarrs_zip",
 ]
 
+[workspace.lints.rust]
+unused_variables = "warn"
+dead_code = "warn"
+missing_docs = "warn"
+unreachable_pub = "warn"
+unsafe_op_in_unsafe_fn = "warn"
+
+[workspace.lints.clippy]
+pedantic = { level = "warn", priority = -1 }
+module_name_repetitions = "allow"
+missing_panics_doc = "warn"
+missing_errors_doc = "warn"
+
 [workspace.dependencies.zarrs_metadata]
-version = "0.2.0"
+version = "0.3.0-dev"
 path = "zarrs_metadata"
 
 [workspace.dependencies.zarrs_storage]
-version = "0.2.2"
+version = "0.3.0"
 path = "zarrs_storage"
 
 [workspace.dependencies.zarrs_filesystem]
-version = "0.1.0"
+version = "0.2.0"
 path = "zarrs_filesystem"
 
 [workspace.dependencies.zarrs_http]
-version = "0.1.0"
+version = "0.2.0"
 path = "zarrs_http"
 
 [workspace.dependencies.zarrs_object_store]
-version = "0.2.1"
+version = "0.3.0"
 path = "zarrs_object_store"
 
 [workspace.dependencies.zarrs_opendal]
-version = "0.3.1"
+version = "0.5.0"
 path = "zarrs_opendal"
 
 [workspace.dependencies.zarrs_zip]
-version = "0.1.0"
+version = "0.2.0"
 path = "zarrs_zip"
 
 [workspace.dependencies.object_store]
 version = "0.11"
 
 [workspace.dependencies.opendal]
-version = "0.50.0"
+version = "0.51.0"
 
 [workspace.dependencies.zip]
 version = "2.1.3"

README.md

@@ -1,24 +1,28 @@
-# zar<u>*rs*</u>
+# zar<ins>rs</ins>
 
 [![Latest Version](https://img.shields.io/crates/v/zarrs.svg)](https://crates.io/crates/zarrs)
 [![zarrs documentation](https://docs.rs/zarrs/badge.svg)](https://docs.rs/zarrs)
 ![msrv](https://img.shields.io/crates/msrv/zarrs)
 [![downloads](https://img.shields.io/crates/d/zarrs)](https://crates.io/crates/zarrs)
 [![build](https://github.com/LDeakin/zarrs/actions/workflows/ci.yml/badge.svg)](https://github.com/LDeakin/zarrs/actions/workflows/ci.yml)
 [![codecov](https://codecov.io/gh/LDeakin/zarrs/graph/badge.svg?token=OBKJQNAZPP)](https://codecov.io/gh/LDeakin/zarrs)
+[![DOI](https://zenodo.org/badge/695021547.svg)](https://zenodo.org/badge/latestdoi/695021547)
 
-`zarrs` is a Rust library for the [Zarr](https://zarr.dev) storage format for multidimensional arrays and metadata. It supports:
- - [Zarr V3](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html), and
- - (New in 0.15) A [V3 compatible subset](https://docs.rs/zarrs/latest/zarrs/#implementation-status) of [Zarr V2](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html).
+`zarrs` is a Rust library for the [Zarr](https://zarr.dev) storage format for multidimensional arrays and metadata. It supports [Zarr V3](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html) and a [V3 compatible subset](https://docs.rs/zarrs/latest/zarrs/#implementation-status) of [Zarr V2](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html).
 
 A changelog can be found [here](https://github.com/LDeakin/zarrs/blob/main/CHANGELOG.md).
 Correctness issues with past versions are [detailed here](https://github.com/LDeakin/zarrs/blob/main/doc/correctness_issues.md).
 
 Developed at the [Department of Materials Physics](https://physics.anu.edu.au/research/mp/), Australian National University, Canberra, Australia.
 
+> [!TIP]
+> If you are a Python user, check out [`zarrs-python`](https://github.com/ilan-gold/zarrs-python).
+> It includes a high-performance codec pipeline for the reference [`zarr-python`](https://github.com/zarr-developers/zarr-python) implementation.
+
 ## Getting Started
 - Review the [implementation status](https://docs.rs/zarrs/latest/zarrs/#implementation-status), [array support](https://docs.rs/zarrs/latest/zarrs/#array-support), and [storage support](https://docs.rs/zarrs/latest/zarrs/#storage-support).
-- View the [examples](https://github.com/LDeakin/zarrs/tree/main/examples) and [the example below](#example).
+- Read [The `zarrs` Book](https://book.zarrs.dev).
+- View the [examples](https://github.com/LDeakin/zarrs/tree/main/zarrs/examples) and [the example below](#example).
 - Read the [documentation](https://docs.rs/zarrs/latest/zarrs/). [`array::Array`](https://docs.rs/zarrs/latest/zarrs/array/struct.Array.html) is a good place to start.
 - Check out the [`zarrs` ecosystem](#zarrs-ecosystem).
 
@@ -90,28 +94,39 @@ println!("{array_ndarray:4}");
 | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
 | **Core**                                                                                      |                                                                                                                                 |
 | [![zarrs_ver]](https://crates.io/crates/zarrs) `zarrs`                                        | [![docs]](https://docs.rs/zarrs)              The core library for manipulating Zarr hierarchies                                |
-| [![zarrs_metadata_ver]](https://crates.io/crates/zarrs_metadata) `zarrs_metadata`             | [![docs]](https://docs.rs/zarrs_metadata)     Zarr metadata support                                                             |
-| [![zarrs_storage_ver]](https://crates.io/crates/zarrs_storage) `zarrs_storage`                | [![docs]](https://docs.rs/zarrs_storage)      The storage API for `zarrs`                                                       |
+| [![zarrs_metadata_ver]](https://crates.io/crates/zarrs_metadata) `zarrs_metadata`             | [![docs]](https://docs.rs/zarrs_metadata)     Zarr metadata support (re-exported as `zarrs::metadata`)                          |
+| [![zarrs_storage_ver]](https://crates.io/crates/zarrs_storage) `zarrs_storage`                | [![docs]](https://docs.rs/zarrs_storage)      The storage API for `zarrs` (re-exported as `zarrs::storage`)                     |
 | **Stores**                                                                                    |                                                                                                                                 |
-| [![zarrs_filesystem_ver]](https://crates.io/crates/zarrs_filesystem) `zarrs_filesystem`       | [![docs]](https://docs.rs/zarrs_filesystem)   A filesystem store                                                                |
+| [![zarrs_filesystem_ver]](https://crates.io/crates/zarrs_filesystem) `zarrs_filesystem`       | [![docs]](https://docs.rs/zarrs_filesystem)   A filesystem store (re-exported as `zarrs::filesystem`)                           |
 | [![zarrs_object_store_ver]](https://crates.io/crates/zarrs_object_store) `zarrs_object_store` | [![docs]](https://docs.rs/zarrs_object_store) [`object_store`](https://docs.rs/object_store/latest/object_store/) store support |
 | [![zarrs_opendal_ver]](https://crates.io/crates/zarrs_opendal) `zarrs_opendal`                | [![docs]](https://docs.rs/zarrs_opendal)      [`opendal`](https://docs.rs/opendal/latest/opendal/) store support                |
 | [![zarrs_http_ver]](https://crates.io/crates/zarrs_http) `zarrs_http`                         | [![docs]](https://docs.rs/zarrs_http)         A synchronous http store                                                          |
 | [![zarrs_zip_ver]](https://crates.io/crates/zarrs_zip) `zarrs_zip`                            | [![docs]](https://docs.rs/zarrs_zip)          A storage adapter for zip files                                                   |
+| [![zarrs_icechunk_ver]](https://crates.io/crates/zarrs_icechunk) [zarrs_icechunk]             | [![docs]](https://docs.rs/zarrs_icechunk)     [`icechunk`](https://docs.rs/icechunk/latest/icechunk/) store support             |
 | **Bindings**                                                                                  |                                                                                                                                 |
+| [![zarrs_python_ver]](https://pypi.org/project/zarrs/) [zarrs-python]                         | [![docs]](https://zarrs-python.readthedocs.io/en/latest/) A codec pipeline for [zarr-python]                                    |
 | [![zarrs_ffi_ver]](https://crates.io/crates/zarrs_ffi) [zarrs_ffi]                            | [![docs]](https://docs.rs/zarrs_ffi)          A subset of `zarrs` exposed as a C/C++ API                                        |
+| **Zarr Metadata Conventions**                                                                 |                                                                                                                                 |
+| [![ome_zarr_metadata_ver]](https://crates.io/crates/ome_zarr_metadata) [ome_zarr_metadata]    | [![docs]](https://docs.rs/ome_zarr_metadata)  A library for OME-Zarr (previously OME-NGFF) metadata                             |
 
 [docs]: https://img.shields.io/badge/docs-brightgreen
-[zarrs_ver]: https://img.shields.io/crates/v/zarrs?label=
-[zarrs_metadata_ver]: https://img.shields.io/crates/v/zarrs_metadata?label=
-[zarrs_storage_ver]: https://img.shields.io/crates/v/zarrs_storage?label=
-[zarrs_filesystem_ver]: https://img.shields.io/crates/v/zarrs_filesystem?label=
-[zarrs_http_ver]: https://img.shields.io/crates/v/zarrs_http?label=
-[zarrs_object_store_ver]: https://img.shields.io/crates/v/zarrs_object_store?label=
-[zarrs_opendal_ver]: https://img.shields.io/crates/v/zarrs_opendal?label=
-[zarrs_zip_ver]: https://img.shields.io/crates/v/zarrs_zip?label=
-[zarrs_ffi_ver]: https://img.shields.io/crates/v/zarrs_ffi?label=
+[zarrs_ver]: https://img.shields.io/crates/v/zarrs
+[zarrs_metadata_ver]: https://img.shields.io/crates/v/zarrs_metadata
+[zarrs_storage_ver]: https://img.shields.io/crates/v/zarrs_storage
+[zarrs_filesystem_ver]: https://img.shields.io/crates/v/zarrs_filesystem
+[zarrs_http_ver]: https://img.shields.io/crates/v/zarrs_http
+[zarrs_object_store_ver]: https://img.shields.io/crates/v/zarrs_object_store
+[zarrs_opendal_ver]: https://img.shields.io/crates/v/zarrs_opendal
+[zarrs_zip_ver]: https://img.shields.io/crates/v/zarrs_zip
+[zarrs_icechunk_ver]: https://img.shields.io/crates/v/zarrs_icechunk
+[zarrs_icechunk]: https://github.com/LDeakin/zarrs_icechunk
+[zarrs_ffi_ver]: https://img.shields.io/crates/v/zarrs_ffi
 [zarrs_ffi]: https://github.com/LDeakin/zarrs_ffi
+[zarrs_python_ver]: https://img.shields.io/pypi/v/zarrs
+[zarrs-python]: https://github.com/ilan-gold/zarrs-python
+[zarr-python]: https://github.com/zarr-developers/zarr-python
+[ome_zarr_metadata_ver]: https://img.shields.io/crates/v/ome_zarr_metadata
+[ome_zarr_metadata]: https://github.com/LDeakin/rust_ome_zarr_metadata
 
 #### [zarrs_tools]
 [![zarrs_tools_ver]](https://crates.io/crates/zarrs_tools) [![zarrs_tools_doc]](https://docs.rs/zarrs_tools)