Answer to the question.
Rust employs a layout optimization strategy known as niche value filling to eliminate the storage overhead of enum discriminants when variants contain types with invalid bit patterns. The compiler identifies "niche" values within a type's representable range—such as the zero value for NonZeroU32 or the null pointer for references—and repurposes these bit patterns to encode other enum variants like None. This transformation relies on the payload type possessing a restricted validity range defined by its intrinsic properties or internal rustc_layout attributes. For a type to serve as a valid niche carrier, it must exhibit at least one bit pattern that constitutes undefined behavior to construct or read, thereby allowing the compiler to reserve that pattern for the enum's alternative variants without allocating additional discriminant space.
Situation from life
While developing a high-frequency trading engine, our team encountered severe cache pressure when storing millions of order timestamps in a Vec<Option<u64>>. Each optional timestamp consumed 16 bytes due to alignment and discriminant overhead, despite the timestamps themselves being strictly positive Unix epoch values. We urgently needed to reduce memory footprint without sacrificing safety or resorting to raw pointers that would complicate the Send and Sync guarantees required for cross-thread processing.
One approach considered was manual bit-packing using raw u64 values and sentinel zero values with unsafe conversion functions. This solution promised maximal memory efficiency but introduced catastrophic risks: a logic error could construct an invalid NonZeroU64 or dereference a null pointer disguised as zero, violating Rust's memory safety invariants. Furthermore, it would require extensive audit trails and unsafe blocks that the team sought to avoid.
Another candidate involved using Optionstd::num::NonZeroU64 directly, leveraging the standard library's guaranteed niche optimization. This approach maintained full type safety and ergonomic match expressions while ensuring the Option occupied exactly 8 bytes instead of 16. The primary constraint was that we had to guarantee timestamps were never zero, which held true for our domain logic since all timestamps were post-1970.
We selected the second solution, refactoring our Timestamp newtype to wrap NonZeroU64 and validating inputs at the system boundary. The result was a 50% reduction in memory usage for our primary order book cache. This optimization eliminated cache thrashing and improved lookup latency by 30%, all achieved without a single line of unsafe code.
What candidates often miss
Why does Option<u32> consume 8 bytes while Option<NonZeroU32> consumes only 4, and how does this optimization behave with nested types like Option<Option<NonZeroU32>>?
The u32 type admits all 2^32 bit patterns as valid, leaving no "spare" bit pattern for the compiler to repurpose as the None variant. Consequently, the compiler must append a discriminant byte (padded to 4 bytes for alignment), yielding 8 bytes total. Conversely, NonZeroU32 explicitly declares that the bit pattern 0x00000000 is invalid, creating a niche that Rust uses to encode None, allowing the resulting Option to occupy exactly 4 bytes.
For nested structures, the optimization chains effectively: Option<Option<NonZeroU32>> remains 4 bytes because the outer Option utilizes a different invalid bit pattern (e.g., 0x00000001) from the available niche space of NonZeroU32. This recursive optimization continues provided the carrier type possesses sufficient invalid bit patterns to accommodate all enum discriminant values.
How do explicit layout attributes like #[repr(C)] or #[repr(u8)] interact with niche optimization, and why does this interaction matter for FFI boundaries?
When applying #[repr(C)] or #[repr(u8)], the programmer mandates a fixed memory layout where the discriminant occupies a specific offset with a defined size. This explicit representation effectively disables niche optimization, ensuring ABI compatibility with C structures that expect explicit tags but forcing the enum to consume additional space for the discriminant.
In FFI contexts, this distinction proves critical because C code expects the discriminant at a predictable, stable offset. Passing a niche-optimized Rust enum lacking explicit repr attributes across the boundary results in undefined behavior, whereas #[repr(C)] guarantees layout stability at the necessary cost of memory efficiency.
What prevents MaybeUninit<T> from serving as a niche carrier for enum optimization even when T itself possesses invalid bit patterns, such as in Option<MaybeUninit<NonZeroU32>>?
MaybeUninit<T> is architecturally designed to hold any bit pattern without invoking undefined behavior, as its purpose is to represent potentially uninitialized memory. Consequently, the compiler treats MaybeUninit<T> as having no invalid bit patterns, meaning its validity range encompasses all 2^(8*sizeof(T)) possible bit combinations. This total validity eliminates any available niches that could be repurposed for enum optimization, regardless of the properties of T.
Therefore, Option<MaybeUninit<NonZeroU32>> occupies 8 bytes—the size of MaybeUninit<u32> plus discriminant padding—despite the underlying NonZeroU32 having restricted validity. This behavior illustrates that niche optimization operates strictly on the immediate type's validity constraints rather than transitive properties of its potential contents.