Review Safe Transmute

[rylev]

Summary

Transmuting values through std::mem::transmute and related APIs is surprisingly unsafe even though the rules for when it is safe to transmute values are fairly straight forward and often verifiable statically. A working group has been established to introduce mechanisms into Rust for safely transmuting values, but progress has stagnated as a consensus on the best way to achieve this cannot be reached.

A topic tangentially related to this, is the question of marker traits which describe the layout of a type for ensuring safety in unsafe code. For instance a Zeroable trait which guarantees its implementers can be zero initialized.

Background reading

Safe transmute has been fairly deeply explored in the Rust ecosystem. There are currently two approaches which are fairly orthogonal which attempt to address this issue, each with their own pros and cons.

Marker Trait Based

The first approach is through using marker traits and associated derive macros to establish certain static properties of a type's layout in memory that can then be used to build safe wrapper functions around the unsafe std::mem::transmute. These marker traits include FixedLayout for types with layouts that can be relied upon and FromBytes for types that can be safely transmuted from an appropriately sized and aligned byte array.

This approach is currently being explored in the mem-markers repo. You can also read more about this approach (albeit from a slightly different angle than mem-markers) in this internals post.

Pros

• Fairly simple and relies on the same mechanisms (namely marker traits, and derive macros) that already employed by the language.
• Provides markers that are useful beyond straight transmute (e.g., specifying whether a type is safe to be zeroed). The other approach mentioned here can cover this use case but in a less straightforward way.

Cons

• Not very fine grained because marker-traits can only expose so much about a trait, and thus the approach is also fairly conservative meaning that there are some transmute operations that are safe that would not be allowed.
• Can't handle lifetime lengthening and shortening (e.g., allowing transmute from 'static to 'a, but not allowing transmute from 'a to 'static)

Type-Level Layout

This approach attempts to model a types layout in the type system and use type checking to prove whether two layout types are equivalent. This approach use type level programming to model a types layout as a trait and then sees if this layout can be transformed into the layout of another type.

This approach is being explored in the typic crate.

Pros

• Strictly more flexible than the marker-trait approach as all the marker traits can be modeled by it, and it can represent transformations that marker traits cannot (e.g., two types have fields of equal offset that are booleans).
• Can handle lifetime lengthening and shortening

Cons

• Much more complex as it relies on type level programming to model type layouts. It should be possible to expose friendly traits for most use cases so that end users are never exposed to how this works, but this is not 100% known yet. Also, the flexibility of this approach may mean having to expose a somewhat complex API to end users even if this is not exposes as type level programming.

Subtleties

The following are various subtleties of the design space that were found to be surprising or not initially considered. We leave them here to ensure they are kept in mind.

• Safe vs. Sound Transmute: Sound transmutes represent legal transmute of memory contents while not necessarily conserving application level invariants. Safe transmutes are sound plus they conserve application level invariants. For instance, transmuting from u16 to u8 is neither safe nor sound while transmuting from u8 to [repr(transparent)] MyNonZerou8(u8) is sound but safe. Having this distinction can be very helpful as developers may wish to check invariants and then before a sound transmute that cannot be statically guaranteed to be safe. This is often up to the user to declare that their type has no invariants.
• Owned vs Reference Transmute: The rules for what is safe to transmute change slightly when dealing with owned transmute to reference transmute. For instance, an owned NonZeroU8 can be turned into a u8 legally, but a &mut NonZeroU8 cannot since others can eventually view this memory as NonZeroU8 and the zero invariant may not have been upheld.

About this issue

This issue corresponds to a lang-team design meeting proposal. It corresponds
to a possible topic of discussion that may be scheduled for deeper discussion
during one of our design meetings.

[nikomatsakis]

Now scheduled for July 1.

[nikomatsakis]

Minutes and recording available