Vec-of-array

[Kromey]

While Vec is Serialize/Deserialize, const generic and large arrays are not, meaning that Vecs containing these can not be handled through Serde. Neither does this library (currently) handle them, since we're unable to apply Serde's traits to these types due to Rust's orphan rule.

We should however be able to implement support for Vec<[T; N]>, as well as [[T; N]; M], which I expect should cover most use cases where these structures are being nested; one example of the former can be seen here.

Unfortunately, it appears that without support from Serde itself, supporting arbitrary levels of nesting is not possible.

[jonasbb]

Unfortunately, it appears that without support from Serde itself, supporting arbitrary levels of nesting is not possible.

That is not true as you can see here. You can arbitrarily nest types here. Both array-in-vec and array-in-array are supported, but also array-in-hashmap, etc. The crate is implemented without any support from serde.

[Kromey]

@jonasbb I think you missed the key word there: arbitrary levels of nesting.

Consider for example a Vec<Vec<Vec<Vec<Vec<Vec<Vec<T>>>>>>>. If T is Serialize, this whole monstrosity is Serialize, because Serde has implemented the necessary traits such that each individual item, whether it's another Vec or the final T, can be serialized recursively.

By contrast, take a [[[[T; M]; N]; O]; P]. Even if T is Serialize, if any of M, N, O, or P is either greater than 32 or is a const generic parameter, this construct as a whole is not Serialize. And due to Rust's orphan rule, only Serde can make them Serialize.

The best the rest of us can do -- whether that's serde_with, serde-big-array, or myself -- is to hand implement serializers to handle specific constructs. But we cannot make these Serialize and therefore we cannot support all possible levels of nesting.

[jonasbb]

I think you missed the key word there: arbitrary levels of nesting.

Yes, I mean arbitrary levels of nesting. As long as the type shape within the serde_as annotation matches the type shape of the real type it will work. Try it, the code even works with const generics. Do try the code with array sizes >32.

#[serde_with::serde_as]
#[derive(serde::Serialize)]
struct Wrapper<T: Serialize, const M: usize, const N: usize, const O: usize, const P: usize>(
    #[serde_as(as = "[[[[_; M]; N]; O]; P]")] [[[[T; M]; N]; O]; P],
);

Code example

fn make_array<T: Copy, const M: usize, const N: usize, const O: usize, const P: usize>(
    t: T,
) -> [[[[T; M]; N]; O]; P] {
    [[[[t; M]; N]; O]; P]
}

fn serialize_nested_arrays<
    T: Serialize + Copy,
    const M: usize,
    const N: usize,
    const O: usize,
    const P: usize,
>(
    t: T,
) {
    let arr: [[[[T; M]; N]; O]; P] = make_array(t);

    #[serde_with::serde_as]
    #[derive(Serialize)]
    struct Wrapper<T: Serialize, const M: usize, const N: usize, const O: usize, const P: usize>(
        #[serde_as(as = "[[[[_; M]; N]; O]; P]")] [[[[T; M]; N]; O]; P],
    );

    let s = serde_json::to_string(&Wrapper(arr)).unwrap();
    println!("{}", s.len());
}

// serialize_nested_arrays::<_, 33, 3, 3, 33>(55);
// Prints 30262

You are right that the nested arrays cannot be made Serialize, but that is not necessary to serialize them with serde. Otherwise, neither serde-big-array, serde_with, serbia, nor this crate would work.

hand implement serializers to handle specific constructs

In a way that is true, but at least in the case of serde_with the hand implementation is limited to an attribute on the field which mostly mirrors the real type anyway.

[Kromey]

Thank you for that clarification and correction! I'm going to have to look deeper into how serde_with does this, at a quick glance they seem to be doing exactly what I was trying to do and running into insurmountable compiler errors due to "conflicting implementations". Clearly they did something different than I tried to do, it will just take some time to dig into it and figure out what.

It seems a great crate (that I had no idea existed until after I published this one), though I really don't like the syntax they use, so if I can find a "cleaner" way to approach it I think my crate can still have something to offer.

[jonasbb]

I can give you some hints where to look. serde_with works with two traits SerializeAs and DeserializeAs. The function signatures are mostly compatible with Serialize/Deserialize. There are two types As and Same which act like adapters to glue the two kinds of traits together. This gluing together is also what the #[serde_as] attribute on the struct does.
You need a SerializeAs implementation for each container type (e.g., array, Vec, BTreeMap, available in src/ser/*). That is analogue to the Serialize implementation in serde`.

The SerializeAs trait is defined "backwards" which makes it in some ways more flexible than Serialize. You can easily add now conversion types, and they will compose well with each other.

The SerializeAs trait, was invented by markazmierczak over here. I adapted and extended them for serde_with though.

[Kromey]

Serialization came together quite quickly, and was beautifully magical. Deserialization is proving to be another story altogether. I believe the issue is that I'm trying to get Rust to infer the generic type parameter based on the function output, and more specifically on an associated type of a trait, but Rust's inference either depends on the function input and/or can't peek into traits' associated types.

I think I'm finally beginning to see why serde_with depends on the syntax it does! I was hoping to come up with a simpler, "cleaner" API by sacrificing some of the awesome power serde_with provides, but I may end up with something just as involved yet far less capable. Still poking at this however, so we'll see where I get to next.

[brainstorm]

Oh, no, I really needed this for my current toy project :_/

/// N encodes password, only 65535 login/password combinations possible
#[derive(Debug, Copy, Clone, Serialize, Deserialize)]
struct LoginPasswordBruteForcer<const N: usize> {
    #[serde(with = "serde_arrays")]
    state: [State; N],
    #[serde(with = "serde_arrays")]
    good_keys: [u16; N],
    #[serde(with = "serde_arrays")]
    bad_keys: [[u16; N]; N],
}

I wanted to serialize/deserialize the login/password attempt "sessions" so that I don't have to start from scratch when re-starting my program, hence bad_keys would store the failed (permission denied) attempts.

The discussion (and work) being done here is a bit over my current Rust skill level, so I'm just asking as a user (kudos for this crate, btw!): will nested Serialization/Deserialization be available anytime soon or Deserialize is still proving elusive, @Kromey?

[Kromey]

@brainstorm I had some personal/family issues come up that suffice to say distracted me from even thinking about this. It has crept back onto my radar recently though, and I'm hoping to tackle this soon. The trick isn't so much implementing nested arrays, but doing so in a manner that won't leave me coded into a corner should I want to expand that support further later.