Rust employs niche value optimization (also called niche filling) to eliminate the discriminant storage for enums when a variant contains a type with invalid bit patterns. For Option<&T>, the None variant is represented by the null pointer value—a bit pattern that is invalid for &T because references must always be non-null. This allows the compiler to store the discriminant implicitly within the pointer itself, ensuring Option<&T> occupies exactly one machine word. This optimization applies to any type possessing niche values—invalid bit patterns such as 0 for NonZeroU32, values outside 0 or 1 for bool, or specific sentinel values in #[repr(C)] structs.
While developing a high-performance abstract syntax tree (AST) for a compiler processing millions of nodes, we faced severe memory pressure from parent and child pointers. Each node required optional references to its parent, left child, and right child, initially implemented as Option<Box<Node>>.
Using Option<Box<Node>> incurred 16 bytes per pointer on 64-bit systems—8 bytes for the Box pointer and 8 bytes for the discriminant and padding. For a tree with 10 million nodes, this amounted to 480 megabytes just for linkage pointers, exceeding our memory budget.
We considered three approaches. First, replacing Option<Box<Node>> with raw pointers (*mut Node) using null for None. This eliminated overhead but required unsafe blocks throughout the codebase, risking dangling pointers and violating Rust's safety guarantees. Second, using an arena allocator with indices (usize) instead of pointers. While cache-friendly, Option<usize> still required 16 bytes due to lack of niches in usize, and index arithmetic complicated the API.
We selected the third approach: Option<NonNull<Node>> wrapped in a safe ParentPtr abstraction. NonNull<T> carries a niche at address 0, allowing Option<NonNull<Node>> to remain 8 bytes. We encapsulated the unsafe dereferencing within the wrapper methods, preserving memory safety while achieving zero-cost abstraction. This reduced the AST memory footprint by 50%, fitting within our 256MB constraint without sacrificing safety.
Why does Option<Option<bool>> remain a single byte while Option<Option<usize>> expands to 16 bytes?
bool possesses 254 niche values because only bit patterns 0 and 1 are valid. The first Option layer consumes one niche (e.g., 2) to represent None, leaving 253 remaining niches. The second Option layer consumes another niche (e.g., 3) for its None variant. Consequently, Option<Option<bool>> still fits within one byte. Conversely, usize has no invalid bit patterns—all 2^64 values are valid memory addresses or data. Without niches, Option<usize> must append a discriminant byte, resulting in 16 bytes (8 for data, 8 for alignment). Nested Option layers cannot optimize further without available niches, so Option<Option<usize>> remains 16 bytes with internal discriminant logic.
Why does the compiler reject niche optimization for enums marked with #[repr(C)] even when the payload type contains niches?
The #[repr(C)] attribute guarantees a C-compatible memory layout with stable field ordering and explicit discriminant storage at a predictable offset. The C language standard does not support overlapping discriminant values with payload data—discriminants must reside in dedicated memory locations to ensure FFI compatibility. While a struct like NonNull<T> contains niches (null pointer), #[repr(C)] enums cannot exploit these to overlap with the discriminant because external C code expects to read a distinct discriminant value at a fixed offset. This restriction preserves interoperability at the cost of memory efficiency, ensuring that sizeof(Option<&T>) equals sizeof(&T) + sizeof(discriminant) under #[repr(C)], typically 16 bytes rather than 8.
How does std::mem::discriminant function for types like Option<&T> that lack explicit discriminant storage in memory?
std::mem::discriminant returns an opaque Discriminant<T> value that uniquely identifies the enum variant regardless of the underlying memory representation. For Option<&T>, the compiler generates code that derives the discriminant by inspecting the pointer value—returning a constant representing Some if the pointer is non-null, and a constant representing None if it is null. Although no separate memory location stores a discriminant tag, the Discriminant type abstracts this computation, allowing variant comparison via == without exposing the niche encoding details. This demonstrates that discriminant operates on semantic variant identity rather than physical memory layout, enabling consistent behavior across optimized and non-optimized enum representations.