Inference15d ago
vLLM Narrows Gap with DeepInfer to 6% on NVIDIA GB200, Reaching 1M Tokens/s per GPU on GB300
Hours after DeepSeek open-sourced DeepInfer, the vLLM project team released official benchmark parity data serving DeepSeek-V4.1-Flash. Utilizing DeepSeek's identical decoding setup (131K context window, nextn=3 speculative decoding), vLLM closed its throughput performance delta to within 6% of DeepInfer across batch sizes 8 through 256 on NVIDIA GB200 systems. On GB300 hardware, vLLM attained up to 1,000,000 tokens/second per GPU and 826 tokens/second per user, generating $38.96 in hourly GPU revenue at DeepSeek off-peak rates. vLLM simultaneously shipped day-0 runtime support for FastH3-Omni and Spark-X2.6-4B.
Tooling16d ago
Eagle releases EAGLE-4, a speculative-decoding drafter that adapts to batch size and hardware; it reports 961.6 tok/s on Qwen3.8-27B for a single request
On Oct 20, inference company Eagle released EAGLE-4, the fourth generation of the EAGLE speculative-decoding family. One drafter resizes its speculative tree for the batch size, hardware and workload while keeping the target model's output distribution. Eagle reports 961.6 tokens/s for one request on Qwen3.8-27B with 4x B300. The code ships as eagle-harness v0.1.0 with plugins for SGLang, vLLM and an experimental TokenSpeed backend. Many reposts call it open source, but the official license is source-available, not OSI open source.
Tooling43d ago
vLLM v0.30.0 ships 762 commits with hot paths for Kimi K3, DeepSeek-V4.1-Flash, Qwen3.8-Flash-Next
The vLLM project (@vllm_project) released v0.30.0 with 762 commits from 315 contributors. Highlights include hybrid-attention hot paths for Kimi K3 (KDA/AttnRes/MLA), DeepSeek-V4.1-Flash (MXFP8 KV and async Engram), and Qwen3.8-Flash-Next (QSA/PLE, lower sparse-GQA overhead); plus HiSparse host-tier decode, Model Runner V2 EAGLE3 pipeline parallelism with adaptive verification, dual-key Gumbel-max watermarking, Fast Start CUDA IPC weight cache, and new models such as GLM-5.3-Flash, K2-Horizon, Cohere Compass, and Bailing V3 VL.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.