Exact Inference

How it works

Highlighted claims

• The two-step sampler is exact because its exit probabilities match the mixture weights. — N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation
• N-vium does not permanently skip upper-layer computation for early-exited tokens. — N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation
• Deferred upper-layer states are processed with later tokens through piggybacking so every token eventually traverses all layers. — N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation
• Inference may either build the full mixture distribution explicitly or sample routing decisions and exit distributions in two steps. — N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation