Feedback requested! 0.8 pre-release

[joshlf]

EDIT: We've released 0.8!

---

This is a draft of our 0.8 release notes. We would love to get your feedback! Please kick the tires on our latest 0.8 alpha release and let us know what's working for you, what's not, what we should change, etc. Any and all feedback is welcome!

These notes accurately describe the state of our current 0.8 alpha releases, but small API changes may still happen before 0.8 is stabilized. If any such changes happen, they will likely just affect method naming.

We are pleased to announce the release of zerocopy 0.8. Zerocopy is a crate that provides safe abstractions for transmutation, and it is the foundation of a wide variety of security-critical systems software — networking stacks, filesystems, cryptograhic firmware, and more — at AWS, Google, and others. In this release, we have extended our safe transmute analysis to support a broader range of types, including:

• dynamically sized types
• types with conditional validity invariants (e.g., bool)
• types containing UnsafeCells

...among other improvements.

Call for Support

Zerocopy has been essential to OpenHCL's success. It has eliminated a myriad of uses of unsafe in code operating on untrusted data, without hurting ergonomics or performance.
— John Starks | Partner Software Engineer at Microsoft, OpenHCL

Success stories like the above both delight us and are crucial to demonstrating to our employeers that zerocopy is a worthy investment of engineering time. If our work has made your life easier, please let us know! Zerocopy's continued development depends on your suppport.

Support for Dynamically Sized Types

Zerocopy now has comprehensive support for slice-based dynamically-sized types (slice DSTs). A slice DST has a fixed prefix followed by a variable number of trailing elements. Slice DSTs are especially useful when modeling packet or file formats with variable-length fields; e.g.:

use zerocopy::*;
use zerocopy_derive::*;

#[derive(FromBytes, KnownLayout, Immutable)]
#[repr(C)]
struct PacketHeader {
    src_port: [u8; 2],
    dst_port: [u8; 2],
    length: [u8; 2],
    checksum: [u8; 2],
}

#[derive(FromBytes, KnownLayout, Immutable)]
#[repr(C)]
struct Packet {
    header: PacketHeader,
    body: [u8],
}

let bytes = &[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11][..];

let packet = Packet::ref_from_bytes(bytes).unwrap();

assert_eq!(packet.header.src_port, [0, 1]);
assert_eq!(packet.header.dst_port, [2, 3]);
assert_eq!(packet.header.length, [4, 5]);
assert_eq!(packet.header.checksum, [6, 7]);
assert_eq!(packet.body, [8, 9, 10, 11]);

Zerocopy supports these types via the new KnownLayout trait, which abstracts over the distinction between Sized types, slice types, and slice DSTs.

Support for Fallible Conversions

Zerocopy now supports conditionally correct transmutations — performing only the minimum set of runtime checks to confirm that the source value is a valid instance of the destination type. For example:

use zerocopy::*;

// 0u8 → bool = false
assert_eq!(try_transmute!(0u8), Ok(false));

// 1u8 → bool = true
assert_eq!(try_transmute!(1u8), Ok(true));

// 2u8 → bool = error
assert!(matches!(
    try_transmute!(3u8),
    Result::<bool, _>::Err(ValidityError { .. })
));

This functionality is made possible by our new, derivable TryFromBytes trait. You can use this trait to model formats with magic numbers; e.g.:

use zerocopy::*;
use zerocopy_derive::*;

// The only valid value of this type is the byte `0xC0`
#[derive(TryFromBytes, KnownLayout, Immutable)]
#[repr(u8)]
enum C0 {
    xC0 = 0xC0,
}

// The only valid value of this type is the byte sequence `0xC0C0`.
#[derive(TryFromBytes, KnownLayout, Immutable)]
#[repr(C)]
struct C0C0(C0, C0);

// The memory layout of a HotChocolatePacket starts with the byte sequence
// `0xC0C0`, followed by mug size (1 bytes) and temperature (1 byte).
#[derive(TryFromBytes, KnownLayout, Immutable)]
#[repr(C)]
struct HotChocolatePacket {
    magic_number: C0C0,
    mug_size: u8,
    temperature: u8,
}

// These bytes correspond to a valid packet.
let bytes = &[0xC0, 0xC0, 240, 77][..];

let packet = HotChocolatePacket::try_ref_from_bytes(bytes).unwrap();
assert_eq!(packet.mug_size, 240);
assert_eq!(packet.temperature, 77);

// These bytes are not a valid packet.
let bytes = &[0x70, 0xC0, 240, 77][..];
assert!(HotChocolatePacket::try_ref_from_prefix(bytes).is_err());

At the time of release, derive(TryFromBytes) works on structs, unions, and enums. Types implementing TryFromBytes can be used as the destination type for invocations our new macros, try_transmute, try_transmute_ref and try_transmute_mut. In our upcoming releases, we plan to add support for custom validators.

Support for UnsafeCells, Atomics

Zerocopy's by-value and by-mut conversions now support types containing UnsafeCells. This enables conversions involving types that permit interior mutability, like atomics; e.g.:

use zerocopy::transmute_mut;
use std::sync::atomic::AtomicU8;

let mut atomic = AtomicU8::new(42);
let primitive: &mut u8 = transmute_mut!(&mut atomic);
assert_eq!(primitive, &mut 42u8);

By-value and by-mut transmutations support types containing UnsafeCells. By-ref transmutations only permit types which do not contain UnsafeCells; this is enforced using the new Immutable trait.

Other Improvements

Return Type Overhaul

Zerocopy's conversion APIs now consistently return Result, with the error type providing both the logical cause of the failure and the source value that precipated the failure. The error type is fine-grained, allowing the user to reason about what error conditions are possible.

This directly supports the common usage pattern of incrementally parsing from a buffer, makes certain error-handling patterns possible, and drastically improves error messages; e.g.:

use zerocopy::{FromBytes, IntoBytes};

let maybe_u16 = u16::read_from_bytes(42u8.as_bytes());

assert_eq!(
    maybe_u16.unwrap_err().to_string(),
    "The conversion failed because the source was incorrectly \
    sized to complete the conversion into the destination type.\n\
    \n\
    Source type: &[u8]\n\
    Source size: 1 byte\n\
    Destination size: 2 bytes\n\
    Destination type: u16"
);

Different APIs have different error conditions; these are now accurately modeled by those APIs' error types:

use core::num::NonZeroU16;
use zerocopy::*;

let bytes = &[4, 2];

match NonZeroU16::try_ref_from_bytes(bytes) {
    Ok(nz) => …,
    // three possible failure modes
    Err(TryCastError::Alignment(err)) => …,
    Err(TryCastError::Size(err)) => …,
    Err(TryCastError::Validity(err)) => …,
};

match NonZeroU16::try_read_from_bytes(bytes) {
    Ok(nz) => …,
    Err(TryReadError::Alignment(i)) => match i {},
    // two possible failure modes
    Err(TryReadError::Size(err)) => …,
    Err(TryReadError::Validity(err)) => …,
};

match u16::read_from_bytes(bytes) {
    Ok(n) => …,
    // one possible failure mode
    Err(SizeError {..}) => …,
};

For types with no alignment requirement, alignment errors are impossible. In these cases, our error types support discarding any alignment error infallibly:

use core::num::NonZeroU8;
use zerocopy::{AlignedTryCastError, TryCastError, TryFromBytes};

let bytes = &[4, 2];

// A `TryCastError` encodes a failure due to alignment, size, or validity.
let err: TryCastError<_, _> = 
    <[NonZeroU8; 2]>::try_ref_from_bytes(bytes).unwrap_err();

// We know that the above cast cannot possibly fail due to alignment issues,
// because `[NonZeroU8; 2]` has an alignment of 1:
match err {
    TryCastError::Alignment(err) => unreachable!(),
    TryCastError::Size(err) => …,
    TryCastError::Validity(err) => …,
}

// However, this introduces a panic path that the optimizer might not
// remove! Instead, let zerocopy rule out this possibility for you.
// 
// An `AlignedTryCastError` encodes failure due to size or validity.
// Because `[NonZeroU8; 2]` is `Unaligned`, we can convert from
// `TryCastError` to `AlignedTryCastError` with `.into()`.
let err: AlignedTryCastError<_, _> = err.into();

// Doing so replaces the `Alignment` error type with `Infallible`,
// which can be dismissed without panicking, like so:
match err {
    TryCastError::Alignment(i) => match i {},
    TryCastError::Size(err) => …,
    TryCastError::Validity(err) => …,
}

API Cleanup

Some traits and methods have been renamed so that our API naming is more consistent. In all cases, the old name remains either as a #[doc(hidden)]/#[deprecated] method or as use NewName as OldName (not pub use; uses of the old name are banned, but generate a helpful compiler error). Notable renames include:

• AsBytes -> IntoBytes
• FromZeroes -> FromZeros
• Some FromBytes methods
• Some Ref methods

Succinct Derives

Previously, our derives required that all derived traits be explicitly named - e.g., #[derive(TryFromBytes, FromZeros, FromBytes)]. As of 0.8, deriving a trait automatically derives any super-traits - e.g., you can now just write #[derive(FromBytes)], and TryFromBytes and FromZeros are derived automatically.

Moreconst-friendly APIs

On the byteorder module's types, some methods are now const. Specifically, from_bytes, to_bytes, new, and get.

Permit external impls of ByteSlice

The ByteSlice trait is no longer sealed; it can be implemented by downstream crates. The same is true of sub-traits such as SplitByteSlice and IntoByteSlice.

Lower MSRV to 1.56

We've reduced our MSRV from 1.60 to 1.56.

Miscellaneous

Putting #[derive(IntoBytes)] on a union now requires passing --cfg zerocopy_derive_union_into_bytes in order to gate against potential future unsoundness. We're actively working with Rust to stabilize the necessary language guarantees to support this in a forwards-compatible way. As part of this effort, we need to know how much demand there is for this feature. If you would like to use IntoBytes on unions, please let us know.

[danielverkamp]

A few notes from a preliminary pass at updating a fairly large project (crosvm) - I just did enough to make it compile and did not attempt to take advantage of any new features or do any major cleanups. Main takeaway: the new version works (our tests pass), woohoo! However, the upgrade required a lot of manual tweaks beyond the expected find-and-replace for the renamed types and functions.

Pretty much every type that previously had AsBytes needed to be adjusted to also derive Immutable and sometimes KnownLayout. Probably the sensible thing to do would have been to replace all of our zerocopy derive(...)s with the full set of zerocopy traits in a consistent order for future-proofing and easier find-and-replace, but I didn't think of that until too late (I wish rustfmt would sort these...)

It might be nice to have a super-trait that pulls in all of the other traits for "normal" structs (Zerocopy? PlainOldData?); although this may not be suitable for all use cases, I believe every single one of our structs that implements any zerocopy traits is compatible with all of them, and the changing list of traits makes for a lot of churn (+/- 1000 lines for our use of zerocopy 0.7 to 0.8, mostly in #[derive(...)] and use zerocopy::... lines). I understand that it is important to get the API right before it's locked down, so hopefully the shuffling of traits is only a pre-1.0 phenomenon. :) (Skimming the bug tracker, I guess this is issue #712.)

The compiler errors when a type is missing Immutable are somewhat misleading, although I'm not sure there's much zerocopy can do about this. An example where the compiler recommends adding a reference rather than implementing Immutable (some were even more confusing than this):

error[E0277]: the trait bound `virtio::wl::CtrlVfdNewDmabuf: zerocopy::Immutable` is not satisfied                                          
   --> devices/src/virtio/wl.rs:317:20                                                                                                      
    |                                                                                                                                       
317 |         .write_obj(ctrl_vfd_new_dmabuf)                                                                                               
    |          --------- ^^^^^^^^^^^^^^^^^^^ the trait `zerocopy::Immutable` is not implemented for `virtio::wl::CtrlVfdNewDmabuf`          
    |          |                                                                                                                            
    |          required by a bound introduced by this call                                                                                  
    |                                                                                                                                       
note: required by a bound in `virtio::descriptor_utils::Writer::write_obj`                                                                  
   --> devices/src/virtio/descriptor_utils.rs:536:25                                                                                        
    |                                                                                                                                       
536 |     pub fn write_obj<T: Immutable + IntoBytes>(&mut self, val: T) -> io::Result<()> {                                                 
    |                         ^^^^^^^^^ required by this bound in `Writer::write_obj`                                                       
help: consider borrowing here                                                                                                               
    |                                                                                                                                       
317 |         .write_obj(&ctrl_vfd_new_dmabuf)                                                                                              
    |                    +                                                                                                                  
317 |         .write_obj(&mut ctrl_vfd_new_dmabuf)                                                                                          
    |                    ++++                                                

Taking the compiler's suggestion results in a different error with a recommendation to remove the reference (back to the original code).

Error types like SizeError do not implement std::error::Error, so the functions like read_from_bytes() that now return a Result can't be used with anyhow::Context, while the previous version that returned Option could. Since Debug and Display are already implemented, I think this should just be a matter of adding impl std::error::Error for SizeError {}, but I didn't actually try this out. It may still not be possible to call .context() due to other constraints (Send + Sync). This is easy enough to work around (assuming the caller doesn't really care about the underlying error), and we didn't use this pattern in many places, so it's not really a big deal for us.

[joshlf]

This is very helpful feedback, thank you! Some of this I've experienced working on upgrading our vendored copy in Fuchsia (especially the code churn), but some is new to me.

I've filed a few issues to track all of this:

• Write 0.8 upgrading guide (and maybe write a cargo fix-like tool) #1295
• Improve error message on missing Immutable #1296
• Make error types usable in more contexts #1297

[joshlf]

Update: error types now implement std::error::Error (#1298); this will be released soon in 0.8.0-alpha.15 (#1462).

[joshlf]

@jswrenn Maybe we should mention the new transmute macros we've added?

[jswrenn]

Done!

[NWPlayer123]

It would be useful to include a TryFromBytes example with a custom 'string' type. I deal with a lot of binary file types that store a string as a u16/u32 length + [u8]/str data, which seems like a perfect use of Dynamically Sized Types. The only problem is that KnownLayout doesn't implement str like it does [u8], so I'm unsure of the best way to go about making that type. It looks like that's still a TODO on #29 / #671?

I would also like to see #158 resolved, but I understand if you can't fit that in for 0.8. I looked and it's since been refactored and inlined google/crosvm@742791d, but would look something like this in 0.8:

pub fn zerocopy_from_reader<R: io::Read, T: FromBytes + IntoBytes + FromZeros>(
    mut read: R,
) -> io::Result<T> {
    let mut out = T::new_zeroed();
    read.read_exact(out.as_mut_bytes())?;
    Ok(out)
}

[joshlf]

Re: str: Does this impl do what you need?

And thanks for letting us know about wanting a reader API! It's likely that it won't be on our short-term roadmap, but it's good to know that there's interest in it.

[NWPlayer123]

Okay, I got it figured out, this is my first attempt using zerocopy on my own project, KnownLayout+TryFromBytes+Immutable with try_ref_from_bytes works. I bring up a reader API since it's also used in nod-rs, which reads Nintendo GameCube and Wii discs.

[russellbanks]

It would be helpful to have a TryFromBytes example, either here or in the docs, for reading an enum that has a multi-byte representation while maintaining a specific endianness.

How can I use TryFromBytes to read an enum (#[repr(u32)]) if I want to maintain a specific endianness?

@jswrenn on Discord:

Explicitly provide the discriminants and use .to_be or .to_le, like so:

#[derive(Debug, KnownLayout, Immutable, IntoBytes, TryFromBytes)]
#[repr(u32)]
pub enum DataStoreVersion {
    /// Version 1 of the data store.
    V1 = 9u32.to_le(),

    /// Version 2 of the data store.
    V2 = 10u32.to_le(),
}

[joshlf]

Good idea!