Speculative Decoding Acceleration
Definition
Inference technique where a smaller draft model or predictive head proposes multiple candidate tokens that are verified in parallel by the target LLM in a single forward pass.
Key Characteristics
- Lossless speedup preserving exact target model token probability distributions.
- Dynamic speculation window adaptation based on acceptance rate.
Applications
- High-throughput LLM serving
- Local model latency reduction
Mentions in Source
- βDynamic speculation tuning with DFlash 2 and batched generation.β β mlx-serve-neural-engine-offload