Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement (de-)serializing for [u8; N] #26

Closed
KizzyCode opened this issue Feb 19, 2021 · 2 comments · Fixed by #28
Closed

Implement (de-)serializing for [u8; N] #26

KizzyCode opened this issue Feb 19, 2021 · 2 comments · Fixed by #28

Comments

@KizzyCode
Copy link

KizzyCode commented Feb 19, 2021

Now that const_generics are stable, I wonder if it is possible to implement support for [u8; N] arrays (serde seems to support them)?

jonasbb added a commit to jonasbb/serde_with that referenced this issue Mar 11, 2021
The `Bytes` type is heavily inspired by `serde_bytes` and ports it to the
serde_as system.

```rust
value: Vec<u8>,
```

Compared to `serde_bytes` these improvements are available

1. Integration with the `serde_as` annotation.
    /cc serde-rs/bytes#14
2. Implementation for arrays of arbitrary size (Rust 1.51+).
    /cc serde-rs/bytes#26
jonasbb added a commit to jonasbb/serde_with that referenced this issue Mar 11, 2021
The `Bytes` type is heavily inspired by `serde_bytes` and ports it to the
serde_as system.

```rust
#[serde_as(as = "Bytes")]
value: Vec<u8>,
```

Compared to `serde_bytes` these improvements are available

1. Integration with the `serde_as` annotation.
    /cc serde-rs/bytes#14
2. Implementation for arrays of arbitrary size (Rust 1.51+).
    /cc serde-rs/bytes#26
jonasbb added a commit to jonasbb/serde_with that referenced this issue Mar 11, 2021
The `Bytes` type is heavily inspired by `serde_bytes` and ports it to the
serde_as system.

```rust
value: Vec<u8>,
```

Compared to `serde_bytes` these improvements are available

1. Integration with the `serde_as` annotation.
    /cc serde-rs/bytes#14
2. Implementation for arrays of arbitrary size (Rust 1.51+).
    /cc serde-rs/bytes#26
jonasbb added a commit to jonasbb/serde_with that referenced this issue Mar 12, 2021
The `Bytes` type is heavily inspired by `serde_bytes` and ports it to the
serde_as system.

```rust
value: Vec<u8>,
```

Compared to `serde_bytes` these improvements are available

1. Integration with the `serde_as` annotation.
    /cc serde-rs/bytes#14
2. Implementation for arrays of arbitrary size (Rust 1.51+).
    /cc serde-rs/bytes#26
bors bot added a commit to jonasbb/serde_with that referenced this issue Mar 12, 2021
277: Add a `Bytes` type for more efficient byte sequences r=jonasbb a=jonasbb

The `Bytes` type is heavily inspired by `serde_bytes` and ports it to the
serde_as system.

```rust
#[serde_as(as = "Bytes")]
value: Vec<u8>,
```

Compared to `serde_bytes` these improvements are available

1. Integration with the `serde_as` annotation.
    /cc serde-rs/bytes#14
2. Implementation for arrays of arbitrary size (Rust 1.51+).
    /cc serde-rs/bytes#26

Co-authored-by: Jonas Bushart <[email protected]>
@PiDelport
Copy link

Here's a stop-gap implementation that can be dropped into existing projects:

mod serde_bytes_array {
    use core::convert::TryInto;

    use serde::de::Error;
    use serde::{Deserializer, Serializer};

    /// This just specializes [`serde_bytes::serialize`] to `<T = [u8]>`.
    pub(crate) fn serialize<S>(bytes: &[u8], serializer: S) -> Result<S::Ok, S::Error>
    where
        S: Serializer,
    {
        serde_bytes::serialize(bytes, serializer)
    }

    /// This takes the result of [`serde_bytes::deserialize`] from `[u8]` to `[u8; N]`.
    pub(crate) fn deserialize<'de, D, const N: usize>(deserializer: D) -> Result<[u8; N], D::Error>
    where
        D: Deserializer<'de>,
    {
        let slice: &[u8] = serde_bytes::deserialize(deserializer)?;
        let array: [u8; N] = slice.try_into().map_err(|_| {
            let expected = format!("[u8; {}]", N);
            D::Error::invalid_length(slice.len(), &expected.as_str())
        })?;
        Ok(array)
    }
}

and used with:

#[serde(with = "serde_bytes_array")]

PiDelport added a commit to ntls-io/nautilus-wallet that referenced this issue Aug 20, 2021
PiDelport added a commit to ntls-io/nautilus-wallet that referenced this issue Aug 20, 2021
PiDelport added a commit to ntls-io/nautilus-wallet that referenced this issue Aug 30, 2021
PiDelport added a commit to ntls-io/nautilus-wallet that referenced this issue Aug 31, 2021
PiDelport added a commit to ntls-io/nautilus-wallet that referenced this issue Aug 31, 2021
PiDelport added a commit to ntls-io/nautilus-wallet that referenced this issue Aug 31, 2021
PiDelport added a commit to ntls-io/nautilus-wallet that referenced this issue Sep 1, 2021
PiDelport added a commit to ntls-io/nautilus-wallet that referenced this issue Sep 1, 2021
PiDelport added a commit to ntls-io/nautilus-wallet that referenced this issue Sep 2, 2021
PiDelport added a commit to ntls-io/nautilus-wallet that referenced this issue Sep 7, 2021
PiDelport added a commit to ntls-io/nautilus-wallet that referenced this issue Sep 7, 2021
PiDelport added a commit to ntls-io/nautilus-wallet that referenced this issue Sep 7, 2021
PiDelport added a commit to ntls-io/nautilus-wallet that referenced this issue Sep 7, 2021
PiDelport added a commit to ntls-io/nautilus-wallet that referenced this issue Sep 7, 2021
* feat(web-server/sgx-wallet-test): enable backtraces

* build(web-server/sgx-wallet): add protected_fs SGX library

* feat(web-server): add wallet operation messages, sealing, endpoint

* feat(web-server/sgx-wallet-impl): serde_bytes_array hack

This works around serde-rs/bytes#26

* feat(web-server/sgx-wallet-impl): add some msgpack debug logging

* fix(web-server): use named string variant config for rmp-serde, too

* feat(web-server): allow setting BIND_ADDR from environment

* feat(web-server): better error reporting, and test updates
jgallagher added a commit to oxidecomputer/omicron that referenced this issue Feb 16, 2023
The primary motivation for this PR is to add an `update_id: Uuid` to the
data passed down to installiator via MGS and the SP, which will allow us
to track progress / completion over the bootstrap network.

While making this change, I realized `Uuid`s were be serialized by
ciborium more efficiently than our `[u8; 32]` arrays, so this PR also
optimizes the serialization of the hash fields: they now always
serialize as exactly 34 bytes, instead of varying between 34 and 66.

Specifically, serde has an optimized path for [serializing byte
sequences](https://docs.rs/serde/latest/serde/trait.Serializer.html#tymethod.serialize_bytes),
which ciborium uses (encoding them as a `bytes` CBOR object). However,
serde's derived `Serialize` implementation for `[T; N]` cannot be
specialized for `[u8; N]`, so it serializes them as arbitrary sequences
(which ciborium then encodes as an `array` CBOR object, within which
bytes may take 1 or 2 bytes on the wire, depending on the value).
[serde_bytes](https://docs.rs/serde_bytes/latest/serde_bytes/) exists to
give an easy way to opt into optimizing byte sequences, but it's only
defined for byte slices and `Vec<u8>`, not `[u8; N]`, so I adapted [this
workaround](serde-rs/bytes#26 (comment)),
which didn't quite work with ciborium. The overall effect is most
visible in the tests: the explicit example is obviously shorter, and the
test that previously checked for a `MAX_SIZE` (because the actual
on-the-wire size was variable) now checks for an exact `EXPECTED_SIZE`
(by using `bytes` instead of `array` we now always serialize to exactly
the size we expect).
@so-schen
Copy link

so-schen commented Dec 9, 2023

I created another crate based on latest version of serde_bytes and add generic byte array support. @dtolnay can be merged into this?
https://crates.io/crates/serde_bytes_ng

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging a pull request may close this issue.

3 participants