Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Saveload custom derive #460

Merged
merged 15 commits into from
Oct 15, 2018
Merged

Saveload custom derive #460

merged 15 commits into from
Oct 15, 2018

Conversation

WaDelma
Copy link
Member

@WaDelma WaDelma commented Sep 8, 2018

Split from #434
A custom derive for IntoSerialize and FromDeserialize named #[derive(Saveload)]. It may seem odd to derive them together, but after spending significant time trying to do it the normal way (one derive per trait) it became clear that due to the requirement of "proxy" types that fill the IntoSerialize::Data and FromDeserialize::Data fields, it made sense to do it this way or you start getting into attribute hell.


This change is Reviewable

UserAB1236872 and others added 2 commits May 31, 2018 00:22
`IntoSerialize` and `FromDeserialize` relied on `Component`
which was somewhat pointless. This wouldn't be a problem, but
due to how proc macros work, this is necessary to create something like
`\#[derive(IntoSerialize)]` because then we can ensure every field,
transitively, is `IntoSerialize`.

For the actual (De)`SerializeComponents` traits, this is changed
into `IntoSerialize+Component` and `FromDeserialize+Component` as
a bound to ensure correctness (it doesn't compile without it anyway).

We can then add `#[derive(Saveload)]` which auto-implements
`IntoSerialize`/`FromDeserialize` for most types (excluding those
with references/pointers, and ones with generics for now) by
automatically constructing a matching proxy "Data" type and converting
all the fields with a recursive call to `into`/`from`.
@zakarumych
Copy link
Member

There is no much sense in deriving only one trait anyway. We could merge them altogether.

@torkleyy
Copy link
Member

torkleyy commented Sep 9, 2018

There is no much sense in deriving only one trait anyway. We could merge them altogether.

I split them earlier, I'm sure there was a reason for that. Deriving two traits at once doesn't seem odd to me.

Copy link
Member

@torkleyy torkleyy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First review, mostly formatting. The missing generic implementation should be rather easy to add, I think we should do that.

I didn't review all the files yet.

@@ -9,8 +9,12 @@ keywords = ["gamedev", "parallel", "specs", "ecs", "derive"]
license = "MIT/Apache-2.0"

[dependencies]
syn = "0.12"
syn = {version="0.12", features=["extra-traits"]}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Formatting / spacing

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, isn't there a newer version already? Might be wrong

Copy link
Member Author

@WaDelma WaDelma Sep 21, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried to update this, but the previous derive code was using synom which was removed from re-exports in newer one and trying to depend on it manually didn't work out of the box. Will need more work to do this.

quote = "0.4"

[dev-dependencies]
specs = { path = "..", features=["serde"] }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above

@@ -0,0 +1,487 @@
//! Contains implementations for #[derive(Saveload)]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing backticks

//! Contains implementations for #[derive(Saveload)]
//! which derives both `IntoSerialize` and `FromDeserialize`.
//!
//! Since this requires a proxy "Data" `struct`/`enum`, these two need
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spacing

//! to be derived together or it begins requiring unwieldy attribute explosion.
//!
//! Currently there is one major limitation to this implementation: it cannot handle
//! generic types such as `struct Foo<T> { x: T, e: Entity }`. I'm not good enough at
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be fixed, shouldn't be too hard.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just starting with the review, is this fixed now?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Forgot to remove push commit that removes that documentation

let derive = match ast.data {
Data::Struct(ref mut data) => saveload_struct(data, &mut ast.ident, &mut ast.generics),
Data::Enum(ref data) => saveload_enum(data, &ast.ident, &ast.generics),
Data::Union(_) => panic!("Unions cannot derive IntoSerialize/FromDeserialize at present"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

won't in the future (missing tag); missing backticks + spacing

}

// Implements all elements of saveload common to structs of any type
// (except unit structs like `struct Foo;` since they're auto-saveload);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't quite understand this sentence.

)),
clause => *clause = Some(parse_quote!(
where MA: ::serde::Serialize + ::serde::de::DeserializeOwned + ::specs::saveload::Marker,
for <'deser> MA: ::serde::Deserialize<'deser>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this necessary? Isn't this guaranteed by DeserializeOwned?

&saveload_fields,
)
} else {
panic!("This derive does not support unit structs (Hint: they automatically derive FromDeserialize and IntoSerialize)");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah now I get the above sentence; both places should describe it more like "every unit struct can derive Serialize and Deserialize, thus has a blanket impl fulfill their IntoSerialize / IntoDeserialize".

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think I wasn't thinking when I wrote this. TBH, we could probably just silently do nothing because of this instead of panicking. I think I decided to be explicit for some reason.

Copy link
Member

@torkleyy torkleyy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good otherwise 👍

panic!("Slices are unsupported, use owned types like Vecs or Arrays instead")
}
Type::Reference(_) => panic!("References are unsupported"),
Type::Ptr(_) => panic!("Raw pointer types are unsupported"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is that?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't want to deal with reference/pointer graphs within serialized components (especially with cycles).

The biggest problem is it's not exactly clear what to even do with types that hold references while deserializing, since you'd need to somehow deserialize the contents of the reference and then pass that handle back to the callsite as well.

The funny thing is, we have a pointer that solves all these problems for us -- Entity, which is just a generational weak pointer. So I see no reason to try and solve these issues when we can tell the user "either write your own impl to explain what you mean by "(de)serialize this pointer" or split this into multiple components and use an Entity ID lookup".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh right. I don't know what I was thinking there.

@@ -0,0 +1,70 @@
// These are mostly just compile tests
#![allow(dead_code)]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please just check the result of the method calls, so we've got some additional coverage and don't need to disable warnings. Would be nice to have some more "exotic" types involving entities, like Vec<Entity>.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the overall thread because this is really important.

use storage::{GenericReadStorage, ReadStorage, WriteStorage};
use world::{Component, EntitiesRes, Entity};

/// Converts a component into its serialized form.
/// Converts a data type (usually a [`Component`]) into its serialized form.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not caused by this PR, but this isn't accurate. It gets converted into its serializable form.

@@ -191,7 +219,7 @@ macro_rules! serialize_components {
M: Marker,
$(
$sto: GenericReadStorage<Component = $comp>,
$comp : IntoSerialize<M>,
$comp : IntoSerialize<M>+Component,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spacing

@torkleyy
Copy link
Member

torkleyy commented Sep 9, 2018

@WaDelma Are you going to be working on this?

@WaDelma
Copy link
Member Author

WaDelma commented Sep 9, 2018

Yeah I'll work on this.

@UserAB1236872
Copy link
Contributor

UserAB1236872 commented Sep 9, 2018

@torkleyy says

Please just check the result of the method calls, so we've got some additional coverage and don't need to disable warnings. Would be nice to have some more "exotic" types involving entities, like Vec.

I can tell you that Vec<Entity> doesn't work right now. I think this info got lost in the split, but back then I said:


Crap, I forgot code like

impl IntoSerialize for C where C: Serialize+Clone vs impl IntoSerialize for Vec<T> where T: IntoSerialize would require specialization to resolve (and even then IDK if specialization would solve this).

Not sure how to solve this from a design standpoint, seems like there are only really two options:

  1. Hand-implement IntoSerialize and FromDeserialize for every native type to avoid a blanket impl, and expect others to #[derive(Saveload)] for their own types.
  2. Extend the derive to have special casing for things like Vec<T> and generate specialized code

I feel like 1 is more painful but probably ultimately more flexible, it seems closer to how serde works, but is also a massive undertaking (not in difficulty so much in time and scale).


I'm still not sure how to fix it.

@torkleyy
Copy link
Member

torkleyy commented Sep 9, 2018

Ah right I just read it. What I don't understand though: Vec<Entity> does not implement Serialize so it should also not implement IntoSerialize. So where do you need specialization here?

@UserAB1236872
Copy link
Contributor

UserAB1236872 commented Sep 9, 2018

Vec<Entity> could probably be done on its own, but Vec<T> such that T may contain an Entity can't.

The reason is Rust can't prove the implementations won't conflict.

@torkleyy
Copy link
Member

torkleyy commented Sep 9, 2018

I don't think that'd be such a big issue.

Suggested solution

  • implement traits for Vec<Entity>
  • other fields can either be solved by
    • creating a newtype with an implementation
    • providing an attribute like #[saveload(data = "FooVecData", into = "into_foo_vec_data", from = "from_foo_vec_data"]

/// ```nobuild
/// fn into<F: FnMut(Entity) -> Option<MA>>(&self, mut ids: F) -> Result<Self::Data, Self::Error> {
/// FooSaveloadData (
/// e: IntoSerialize::into(&self.0, &mut ids)?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tuple struct, shouldn't have a name here.

@WaDelma
Copy link
Member Author

WaDelma commented Sep 14, 2018

I stumbled into problem while trying to make custom derive to work with generics: I need to assert that <T as FromDeserialize<MA>>::Data = <T as IntoSerialize<MA>>::Data which triggers error: equality constraints are not yet supported in where clauses (#20041).

So we would need to either wait for that to stabilize, merge the traits or figure some other workaround for it.

This is my current macro expansion

@UserAB1236872
Copy link
Contributor

Merging the traits seems adequate, to me. I don't see much near-term utility in having them split with how all the saveload stuff currently works (as opposed to serde where there can be good reasons to implement only one of (De)serialize). If somebody has a compelling reason they can be split again.

@WaDelma
Copy link
Member Author

WaDelma commented Sep 21, 2018

After merging the traits and getting code gen work I tried to implement ConvertSaveload (I am not fond of the name) for Vec<Entity>, but it cannot be done:

error[E0119]: conflicting implementations of trait `saveload::ConvertSaveload<_>` for type `std::vec::Vec<world::entity::Entity>`:
   --> src/saveload/mod.rs:187:1
    |
143 | / impl<C, M> ConvertSaveload<M> for C
144 | | where
145 | |     C: Clone + Serialize + DeserializeOwned,
146 | | {
...   |
162 | |     }
163 | | }
    | |_- first implementation here
...
187 | / impl<M> ConvertSaveload<M> for Vec<Entity>
188 | | where
189 | |     M: Serialize + DeserializeOwned,
190 | | {
...   |
206 | |     }
207 | | }
    | |_^ conflicting implementation for `std::vec::Vec<world::entity::Entity>`
    |
    = note: upstream crates may add new impl of trait `serde::Serialize` for type `std::vec::Vec<world::entity::Entity>` in future versions
    = note: upstream crates may add new impl of trait `serde::Deserialize<'de>` for type `std::vec::Vec<world::entity::Entity>` in future versions

@UserAB1236872
Copy link
Contributor

Looks like since in the future they're allowing upstream impls of Serialize and Deserialize there could possible be a blanket impl conflict because, if an upstream crate did it, then suddenly the impl<C> would work? That's... bizarre and unfortunate.

@UserAB1236872
Copy link
Contributor

I've checked and annoyingly, even specialization doesn't solve this.

@zakarumych
Copy link
Member

Can addition of C: Component fix this?

@WaDelma
Copy link
Member Author

WaDelma commented Sep 23, 2018

Adding Component bound seems to fix it, but I am not sure if it's right trait here.

Real solution would be adding bound for:

auto trait NoEntity {}

impl !NoEntity for Entity {}

but auto traits are unstable so that's out of the picture

@zakarumych
Copy link
Member

I see. This trait is for deriving.
I have an idea.

@torkleyy
Copy link
Member

Since this seems rather experimental, please move this into an optional crate / hide it behind a feature flag. If this only works for nightly, that would also be a way to get a first implementation.

@WaDelma
Copy link
Member Author

WaDelma commented Sep 23, 2018

I was going to leave it out of this PR anyways. Is there still something missing here otherwise?

/// [`Clone`], [`Serialize`] and [`DeserializeOwned`].
///
/// Implementing this yourself is usually only needed if you
/// have a component that points to another Entity, or has a field which does,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Entity -> [Entity] (and subsequent link at the bottom)

/// and you wish to [`Serialize`] it.
///
/// *Note*: if you're using `specs_derive`
/// and your struct does not have a generic bound (i.e. `struct Foo<T>`),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The generic bound thing was fixed, so we can remove this part.

Copy link
Member

@torkleyy torkleyy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1 of 2 files at r2, 3 of 6 files at r5, 1 of 1 files at r6, 1 of 1 files at r7.
Reviewable status: 6 of 7 files reviewed, 8 unresolved discussions (waiting on @torkleyy, @Zaerei, and @WaDelma)


specs-derive/Cargo.toml, line 16 at r2 (raw file):

Previously, torkleyy (Thomas Schaller) wrote…

Same as above

Done


specs-derive/src/impl_saveload.rs, line 1 at r2 (raw file):

Previously, torkleyy (Thomas Schaller) wrote…

missing backticks

Done


specs-derive/src/impl_saveload.rs, line 4 at r2 (raw file):

Previously, torkleyy (Thomas Schaller) wrote…

spacing

Done.


specs-derive/src/impl_saveload.rs, line 56 at r2 (raw file):

Previously, torkleyy (Thomas Schaller) wrote…

won't in the future (missing tag); missing backticks + spacing

Done.


specs-derive/src/impl_saveload.rs, line 112 at r2 (raw file):

Previously, torkleyy (Thomas Schaller) wrote…

I don't quite understand this sentence.

Done.


specs-derive/src/impl_saveload.rs, line 129 at r2 (raw file):

Previously, torkleyy (Thomas Schaller) wrote…

Why is this necessary? Isn't this guaranteed by DeserializeOwned?

OK, now invalid.


specs-derive/src/impl_saveload.rs, line 479 at r2 (raw file):

Previously, torkleyy (Thomas Schaller) wrote…

Oh right. I don't know what I was thinking there.

OK, nothing left here.


specs-derive/src/impl_saveload.rs, line 117 at r8 (raw file):

        .cloned()
        .map(|mut f| {
            replace_entity_type(&mut f.ty);

In the future, it would be nice if we had a compiletest that checks if a serde derive would do it, too.


specs-derive/tests/saveload.rs, line 9 at r8 (raw file):

#[derive(ConvertSaveload)]
struct OneFieldNamed {
    e: ::specs::Entity,

Please import the prelude.


specs-derive/tests/saveload.rs, line 68 at r8 (raw file):

        // so no need to test anything but unit
        black_box::<U64Marker, _>(AnEnum::Unit);
        

whitespace


src/saveload/de.rs, line 156 at r8 (raw file):

/// ```
///
pub trait FromDeserialize<M>: Component {

That's what I'm a bit more skeptical about.. There was no split before the overhaul, then I added it for some reason (don't remember), now I'm not sure why I did that. And yeah, maybe we can find a better trait name.


src/saveload/mod.rs, line 58 at r7 (raw file):

Previously, Zaerei (Zoe Juozapaitis) wrote…

Entity -> [Entity] (and subsequent link at the bottom)

Did you check the links actually work? It would be good if all of them were absolute, since relative ones break for re-exports. But then, versions might differ..


src/saveload/mod.rs, line 62 at r7 (raw file):

Previously, Zaerei (Zoe Juozapaitis) wrote…

The generic bound thing was fixed, so we can remove this part.

Is this resolved?


src/saveload/ser.rs, line 12 at r2 (raw file):

Previously, torkleyy (Thomas Schaller) wrote…

Not caused by this PR, but this isn't accurate. It gets converted into its serializable form.

OK, no longer valid.


src/saveload/ser.rs, line 222 at r2 (raw file):

Previously, torkleyy (Thomas Schaller) wrote…

spacing

This is still an issue.

@WaDelma
Copy link
Member Author

WaDelma commented Sep 26, 2018

"Please import the prelude.":
I removed import for that to make sure all things referred in custom derive use full path (previously there were inconsistencies here because prelude was used).
I am not sure if I should change everything to work like in #467 though

re relative paths:
Yeah I checked that they work. I think I can actually use trait.Component.html to refer them if they are in same crate

@WaDelma
Copy link
Member Author

WaDelma commented Sep 28, 2018

Absolute links also break if the user is using locally generated documentation

@torkleyy
Copy link
Member

Reviewable still not working for you?

Please ping me when I should review again.

@WaDelma
Copy link
Member Author

WaDelma commented Oct 1, 2018

@torkleyy Should I change things inside the custom derive to work like in #467?
That and concern on the merging of trait/renaming it are only things left.

@ldesgoui ldesgoui mentioned this pull request Oct 2, 2018
3 tasks
@torkleyy
Copy link
Member

torkleyy commented Oct 2, 2018

@WaDelma I don't know what you mean by that rn. I'll review both PRs tomorrow if possible.

@WaDelma
Copy link
Member Author

WaDelma commented Oct 3, 2018

Currently the custom derive does ::specs::Entity, but in that PR the other custom derive was changed to do Entity so you have to import everything the derive needs. In exchange you get ability to (easily) use without direct specs dependency. (You can still make top level module named specs I think)

@torkleyy
Copy link
Member

torkleyy commented Oct 3, 2018 via email

Copy link
Member

@torkleyy torkleyy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 1 of 2 files at r2, 1 of 1 files at r8, 1 of 1 files at r9, 2 of 4 files at r10, 3 of 3 files at r11.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @Zaerei)

Copy link
Member

@torkleyy torkleyy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @torkleyy and @Zaerei)


specs-derive/src/impl_saveload.rs, line 241 at r3 (raw file):

Previously, Zaerei (Zoe Juozapaitis) wrote…

Tuple struct, shouldn't have a name here.

Not sure what this one is about.


src/saveload/mod.rs, line 58 at r7 (raw file):

Previously, torkleyy (Thomas Schaller) wrote…

Did you check the links actually work? It would be good if all of them were absolute, since relative ones break for re-exports. But then, versions might differ..

OK, let's ignore this.


src/saveload/mod.rs, line 62 at r7 (raw file):

Previously, torkleyy (Thomas Schaller) wrote…

Is this resolved?

LGTM

@UserAB1236872
Copy link
Contributor


specs-derive/src/impl_saveload.rs, line 241 at r3 (raw file):

Previously, torkleyy (Thomas Schaller) wrote…

Not sure what this one is about.

It's just a minor documentation error. We have a tuple struct, i.e. FooData(a,b,c) but in the example we're instantiating it like FooData(a: a, b: b, c: c) which doesn't make sense.

@WaDelma
Copy link
Member Author

WaDelma commented Oct 15, 2018

bors r+

Ah yeah, this PR.

bors bot added a commit that referenced this pull request Oct 15, 2018
460: Saveload custom derive r=WaDelma a=WaDelma

Split from #434
A custom derive for IntoSerialize and FromDeserialize named #[derive(Saveload)]. It may seem odd to derive them together, but after spending significant time trying to do it the normal way (one derive per trait) it became clear that due to the requirement of "proxy" types that fill the IntoSerialize::Data and FromDeserialize::Data fields, it made sense to do it this way or you start getting into attribute hell.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/slide-rs/specs/460)
<!-- Reviewable:end -->


Co-authored-by: Zoe Juozapaitis <jragonmiris@gmail.com>
Co-authored-by: Delma <delma@del.ma>
@bors
Copy link
Contributor

bors bot commented Oct 15, 2018

Build succeeded

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants