SGLang v0.5.21 Unveils Dynamic Prefill-Decode Switching and Native Rust Prefix Caching for Extreme LLM Serving
SGLang, a leading open-source high-throughput inference engine for large models, officially releases milestone version v0.5.21, merging 779 PRs across 227 contributors. The release introduces dynamic on-the-fly role switching between Prefill and Decode (PD) instances without restarting, an upgraded high-performance Rust core for the default prefix cache, and 22% faster first-token generation for DeepSeek-V4.1 on long prompts. It also debuts the /v1/decisions and /v1/score APIs, alongside complete support for next-gen models including MiMo-V2.6, DiffusionGemma, and GLM-5.3-Flash on AMD MI355X.
Key Takeaways
- Pioneers zero-restart dynamic role switching between prefill and decode instances to eliminate cluster capacity skew
- Upgrades default prefix cache to a native Rust core, cutting DeepSeek-V4.1 TTFT on long prompts by 22%
- Improves Kimi K3 prefill throughput by 20.6% and adds native AMD MI355X support with FP8/MXFP4 MoE kernels
Linked Models & Agents:
⚡ DeepSeek-V4.1⚡ Kimi K3⚡ GLM-5.3-Flash⚡ MiMo-V2.6🤖 SGLang🤖 vLLM🤖 Radix-Attention🌐 LMSYS🌐 AMD🌐 Moonshot AI