Semver guarantees for data layouts

[the8472]

Spun off from #36 (comment) and previous.

Currently layout guarantees either come from explicit choices (adding repr) or follow from other guarantees (e.g. all-pub struct). #36 aims to introduce a new compiler guarantee that automatically applies to some repr(Rust) structs and would make field reordering a breaking change for unsafe code relying on that compiler guarantee.

The question is whether unsafe code could reasonably expect these guarantees to be upheld by other crates across minor version bumps.

If yes this would place a new requirement on crate authors and those requirements can be subtle if a struct contains generic fields which would make it sometimes homogeneous and sometimes not.

If no then crate authors would have to explicitly provide this guarantee either via comment or some annotation that can be checked by the compiler or semver linters and unsafe code may need compile-time checks that its assumptions still remain true. There also is the risk that unsafe code authors may conflate compiler guarantees with semver guarantees.

[RalfJung]

So the general question here is: when semver guarantees that old clients of the library keep working, to what extend does that apply to clients using unsafe? My personal expectation is "not at all", i.e., semver only covers things that are observable by safe code.

This question lives on the intersection of UCG and... whatever group is responsible for ironing out semver. Wasn't there some library API design group?

[RalfJung]

Cc @rust-lang/libs this is a UCG question that overlaps with semver / API design questions. Your input would be appreciated.

[dtolnay]

Is the concern that someone takes a library type with private fields and transmutes to/from [T; N] to access the fields?

They should not do that. If the fields are private you don't get to assume what they are.

[RalfJung]

Is the concern that someone takes a library type with private fields and transmutes to/from [T; N] to access the fields?

I think that was @the8472' concern, yes.

They should not do that. If the fields are private you don't get to assume what they are.

Agreed. Is there some place this is written down or so? It seems to be worth documenting, given that the question was raised.

[the8472]

No, not private fields. Even a repr(Rust) type with all-pub fields currently does not guarantee its ordering thus it wouldn't be safe for unsafe code to make assumptions about field ordering. #36 would change that, the question is whether that also means unsafe code authors could automatically rely on that to be covered by semver guarantees, i.e. whether crate authors have to start preserving the order because someone else might use it in unsafe code.

[Mark-Simulacrum]

It's probably worth noting that the safe transmute work is currently planning on providing traits which allow libraries to indicate how much can be assumed by consumers, see rust-lang/project-safe-transmute#5 for the latest RFC draft.

[the8472]

The RFC provides a safe interface for transmutation, which is great but does not necessarily answer what authors of unsafe code can assume.

So the general question here is: when semver guarantees that old clients of the library keep working, to what extend does that apply to clients using unsafe? My personal expectation is "not at all", i.e., semver only covers things that are observable by safe code.

For an all-pub struct field offsets are observable from safe code, that doesn't guarantee that those offsets are stable.

[RalfJung]

For an all-pub struct field offsets are observable from safe code, that doesn't guarantee that those offsets are stable.

Fair. Just like type sizes are typically considered excluded from semver (as far as I know) even though they are observable by safe code, I would think the same applies to field offsets.