MLX-Serve: Neural Engine Prefill Offload & Speculative Decoding
Primary Source: https://github.com/ddalcu/mlx-serve/releases/tag/v26.8.10-pre-release.1
Author: ddalcu · Year: 2026 · Publisher: ddalcu
Summary
A high-performance local inference server for Apple Silicon that leverages the Apple Neural Engine (ANE) for prompt prefill while dispatching autoregressive token decode to the GPU, unlocking unprecedented local throughput.
Key Points
- Heterogeneous compute scheduling: ANE handles compute-heavy prompt evaluation while GPU handles memory-bound token decoding.
- Dynamic speculation tuning with DFlash 2 and batched generation.
- Significant energy and memory bandwidth savings on M-series Mac hardware.