Announcing zerocopy 0.8!

[joshlf]

We are pleased to announce the release of zerocopy 0.8 (crate documentation). Zerocopy is a crate that provides safe abstractions for transmutation, and it is the foundation of a wide variety of security-critical systems software — networking stacks, filesystems, cryptograhic firmware, and more — at AWS, Google, and others. In this release, we have extended our safe transmute analysis to support a broader range of types, including:

• dataful enums
• dynamically sized types
• types with conditional validity invariants (e.g., bool)
• types containing UnsafeCells

...among other improvements.

Call for Support

Zerocopy has been essential to our project's success. It has eliminated a myriad of uses of unsafe in code operating on untrusted data, without hurting ergonomics or performance.
— John Starks | Partner Software Engineer at Microsoft, OpenHCL

Success stories like the above both delight us and are crucial to demonstrating to our employeers that zerocopy is a worthy investment of engineering time. If our work has made your life easier, please let us know! Zerocopy's continued development depends on your suppport.

Support for Dataful Enums

Zerocopy now has support for dataful enums; e.g.:

use zerocopy::*;
use zerocopy_derive::*;

#[derive(KnownLayout, Immutable, TryFromBytes)]
#[repr(u32)]
enum Cuppa {
    Tea {
        iced: bool,
        milk: u8,
    } = u32::from_ne_bytes(*b"CHAI"),
    Coco {
        marshmallows: u8,
    } = u32::from_ne_bytes(*b"COCO"),
}

// These bytes correspond to a valid, aligned cup of tea.
let data = u64::from_ne_bytes([b'C', b'H', b'A', b'I', 1, 42, 0, 0]);

assert!(matches!(
    Cuppa::try_ref_from_bytes(data.as_bytes()),
    Ok(Cuppa::Tea {
        iced: true,
        milk: 42,
    })
));

// These bytes correspond to a valid, aligned cup of hot coco.
let data = u64::from_ne_bytes([b'C', b'O', b'C', b'O', 4, 0, 0, 0]);

assert!(matches!(
    Cuppa::try_ref_from_bytes(data.as_bytes()),
    Ok(Cuppa::Coco {
        marshmallows: 4,
    })
));

Support for Dynamically Sized Types

Zerocopy now has comprehensive support for slice-based dynamically-sized types (slice DSTs). A slice DST has a fixed prefix followed by a variable number of trailing elements. Slice DSTs are especially useful when modeling packet or file formats with variable-length fields; e.g.:

use zerocopy::*;
use zerocopy_derive::*;

#[derive(FromBytes, KnownLayout, Immutable)]
#[repr(C)]
struct PacketHeader {
    src_port: [u8; 2],
    dst_port: [u8; 2],
    length: [u8; 2],
    checksum: [u8; 2],
}

#[derive(FromBytes, KnownLayout, Immutable)]
#[repr(C)]
struct Packet {
    header: PacketHeader,
    body: [u8],
}

let bytes = &[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11][..];

let packet = Packet::ref_from_bytes(bytes).unwrap();

assert_eq!(packet.header.src_port, [0, 1]);
assert_eq!(packet.header.dst_port, [2, 3]);
assert_eq!(packet.header.length, [4, 5]);
assert_eq!(packet.header.checksum, [6, 7]);
assert_eq!(packet.body, [8, 9, 10, 11]);

Zerocopy supports these types via the new KnownLayout trait, which abstracts over the distinction between Sized types, slice types, and slice DSTs.

Support for Fallible Conversions

Zerocopy now supports conditionally correct transmutations — performing only the minimum set of runtime checks to confirm that the source value is a valid instance of the destination type. For example:

use zerocopy::*;

// 0u8 → bool = false
assert_eq!(try_transmute!(0u8), Ok(false));

// 1u8 → bool = true
assert_eq!(try_transmute!(1u8), Ok(true));

// 2u8 → bool = error
assert!(matches!(
    try_transmute!(3u8),
    Result::<bool, _>::Err(ValidityError { .. })
));

This functionality is made possible by our new, derivable TryFromBytes trait. You can use this trait to model formats with magic numbers; e.g.:

use zerocopy::*;
use zerocopy_derive::*;

// The only valid value of this type is the byte `0xC0`
#[derive(TryFromBytes, KnownLayout, Immutable)]
#[repr(u8)]
enum C0 {
    xC0 = 0xC0,
}

// The only valid value of this type is the byte sequence `0xC0C0`.
#[derive(TryFromBytes, KnownLayout, Immutable)]
#[repr(C)]
struct C0C0(C0, C0);

// The memory layout of a HotChocolatePacket starts with the byte sequence
// `0xC0C0`, followed by mug size (1 bytes) and temperature (1 byte).
#[derive(TryFromBytes, KnownLayout, Immutable)]
#[repr(C)]
struct HotChocolatePacket {
    magic_number: C0C0,
    mug_size: u8,
    temperature: u8,
}

// These bytes correspond to a valid packet.
let bytes = &[0xC0, 0xC0, 240, 77][..];

let packet = HotChocolatePacket::try_ref_from_bytes(bytes).unwrap();
assert_eq!(packet.mug_size, 240);
assert_eq!(packet.temperature, 77);

// These bytes are not a valid packet.
let bytes = &[0x70, 0xC0, 240, 77][..];
assert!(HotChocolatePacket::try_ref_from_prefix(bytes).is_err());

At the time of release, derive(TryFromBytes) works on structs, unions, and enums. Types implementing TryFromBytes can be used as the destination type for invocations our new macros, try_transmute, try_transmute_ref and try_transmute_mut. In our upcoming releases, we plan to add support for custom validators.

Support for UnsafeCells, Atomics

Zerocopy's by-value and by-mut conversions now support types containing UnsafeCells. This enables conversions involving types that permit interior mutability, like atomics; e.g.:

use zerocopy::transmute_mut;
use std::sync::atomic::AtomicU8;

let mut atomic = AtomicU8::new(42);
let primitive: &mut u8 = transmute_mut!(&mut atomic);
assert_eq!(primitive, &mut 42u8);

By-value and by-mut transmutations support types containing UnsafeCells. By-ref transmutations only permit types which do not contain UnsafeCells; this is enforced using the new Immutable trait.

Other Improvements

Return Type Overhaul

Zerocopy's conversion APIs now consistently return Result, with the error type providing both the logical cause of the failure and the source value that precipated the failure. The error type is fine-grained, allowing the user to reason about what error conditions are possible.

This directly supports the common usage pattern of incrementally parsing from a buffer, makes certain error-handling patterns possible, and drastically improves error messages; e.g.:

use zerocopy::{FromBytes, IntoBytes};

let maybe_u16 = u16::read_from_bytes(42u8.as_bytes());

assert_eq!(
    maybe_u16.unwrap_err().to_string(),
    "The conversion failed because the source was incorrectly \
    sized to complete the conversion into the destination type.\n\
    \n\
    Source type: &[u8]\n\
    Source size: 1 byte\n\
    Destination size: 2 bytes\n\
    Destination type: u16"
);

Different APIs have different error conditions; these are now accurately modeled by those APIs' error types:

use core::num::NonZeroU16;
use zerocopy::*;

let bytes = &[4, 2];

match NonZeroU16::try_ref_from_bytes(bytes) {
    Ok(nz) => …,
    // three possible failure modes
    Err(TryCastError::Alignment(err)) => …,
    Err(TryCastError::Size(err)) => …,
    Err(TryCastError::Validity(err)) => …,
};

match NonZeroU16::try_read_from_bytes(bytes) {
    Ok(nz) => …,
    Err(TryReadError::Alignment(i)) => match i {},
    // two possible failure modes
    Err(TryReadError::Size(err)) => …,
    Err(TryReadError::Validity(err)) => …,
};

match u16::read_from_bytes(bytes) {
    Ok(n) => …,
    // one possible failure mode
    Err(SizeError {..}) => …,
};

For types with no alignment requirement, alignment errors are impossible. In these cases, our error types support discarding any alignment error infallibly:

use core::num::NonZeroU8;
use zerocopy::{AlignedTryCastError, TryCastError, TryFromBytes};

let bytes = &[4, 2];

// A `TryCastError` encodes a failure due to alignment, size, or validity.
let err: TryCastError<_, _> = 
    <[NonZeroU8; 2]>::try_ref_from_bytes(bytes).unwrap_err();

// We know that the above cast cannot possibly fail due to alignment issues,
// because `[NonZeroU8; 2]` has an alignment of 1:
match err {
    TryCastError::Alignment(err) => unreachable!(),
    TryCastError::Size(err) => …,
    TryCastError::Validity(err) => …,
}

// However, this introduces a panic path that the optimizer might not
// remove! Instead, let zerocopy rule out this possibility for you.
// 
// An `AlignedTryCastError` encodes failure due to size or validity.
// Because `[NonZeroU8; 2]` is `Unaligned`, we can convert from
// `TryCastError` to `AlignedTryCastError` with `.into()`.
let err: AlignedTryCastError<_, _> = err.into();

// Doing so replaces the `Alignment` error type with `Infallible`,
// which can be dismissed without panicking, like so:
match err {
    TryCastError::Alignment(i) => match i {},
    TryCastError::Size(err) => …,
    TryCastError::Validity(err) => …,
}

API Cleanup

Some traits and methods have been renamed so that our API naming is more consistent. In all cases, the old name remains either as a #[doc(hidden)]/#[deprecated] method or as use NewName as OldName (not pub use; uses of the old name are banned, but generate a helpful compiler error). Notable renames include:

• AsBytes -> IntoBytes
• FromZeroes -> FromZeros
• Some FromBytes methods
• Some Ref methods

Succinct Derives

Previously, our derives required that all derived traits be explicitly named - e.g., #[derive(TryFromBytes, FromZeros, FromBytes)]. As of 0.8, deriving a trait automatically derives any super-traits - e.g., you can now just write #[derive(FromBytes)], and TryFromBytes and FromZeros are derived automatically.

More const-friendly APIs

On the byteorder module's types, some methods are now const. Specifically, from_bytes, to_bytes, new, and get.

Permit external impls of ByteSlice

The ByteSlice trait is no longer sealed; it can be implemented by downstream crates. The same is true of sub-traits such as SplitByteSlice and IntoByteSlice.

Lower MSRV to 1.56

We've reduced our MSRV from 1.60 to 1.56.

Upgrading guide

Our new 0.8 release includes a number of changes which are backwards-incompatible with 0.7. This guide describes these changes and gives advice for how to make the upgrade process as painless as possible.

 

For large codebases, it may help to automate some of these upgrading steps using a tool such as ast-grep.

Implied derives

The FromBytes trait is a sub-trait of FromZeros. On 0.7, in order to derive FromBytes, you also had to derive FromZeros explicitly:

#[derive(FromZeros, FromBytes)]
struct Foo { ... }

On 0.8, deriving a trait automatically derives its super-traits. This code will now fail to compile since the FromBytes derive will emit an impl of FromZeros, which will conflict with the impl derived by the explicit FromZeros derive. Instead, simply write:

#[derive(FromBytes)]
struct Foo { ... }

New names

The AsBytes trait has been renamed to IntoBytes. The FromZeroes trait has been renamed to FromZeros.

A number of Ref, FromBytes, and IntoBytes methods have been renamed or deprecated in favor of a more general method. The old methods are preserved and marked as #[doc(hidden)] #[deprecated], including a deprecation message that indicates which API should be preferred in 0.8. We recommend using these compilation warnings as a guide for how to upgrade individual call sites. As they are #[doc(hidden)], we do not consider these methods subject to semver guarantees, and they may be removed or re-used in any future 0.8 version.

New traits

FromBytes and IntoBytes

Some properties that were previously required by FromBytes or IntoBytes (previously AsBytes) have now been split into separate traits. APIs that still require those properties now add bounds on these new traits.

The new traits in question are KnownLayout, which encodes certain layout information about a type, and Immutable, which indicates that a type does not contain any UnsafeCells.

We recommend that you simply derive KnownLayout and Immutable on any type with existing zerocopy trait derives. Alternatively, for a more fine-grained approach, use compilation errors as a guide to which types should derive these traits.

ByteSlice

The ByteSlice trait now supports byte slices which cannot be cheaply split in two. The new SplitByteSlice trait extends ByteSlice with this behavior. In 0.8, SplitByteSlice is analogous to what ByteSlice was in 0.7. The same holds for ByteSliceMut and SplitByteSliceMut.

If you don't care about maximum flexibility, we recommend simply replacing all instances of ByteSlice[Mut] with SplitByteSlice[Mut]. Alternatively, if you wish to support ByteSlice[Mut] types which cannot be split, use compilation errors as a guide to where you need to add SplitByteSlice[Mut] bounds.

New return values

Many methods that previously returned Option now return Result, and include a finer-grained error type. If you wish to preserve the 0.7 behavior and minimize the code change needed to upgrade, simply add .ok() to these call sites in order to convert Results back into Options. However, we recommend that most code should try to migrate to the new error types eventually, as they will help you ensure that you're only handling errors that you expect. For example, having to handle an AlignmentError where you don't expect it is an indication that you could use a different API that guarantees freedom from alignment errors (e.g. one of the Ref methods with unaligned in the method name).

IntoBytes support for unions

Putting #[derive(IntoBytes)] on a union now requires passing --cfg zerocopy_derive_union_into_bytes in order to gate against potential future unsoundness. We're actively working with Rust to stabilize the necessary language guarantees to support this in a forwards-compatible way. As part of this effort, we need to know how much demand there is for this feature. If you would like to use IntoBytes on unions, please let us know.

byteorder feature

The byteorder feature has been removed; in 0.8, the byteorder module is always present, and is not optional.