Inference16d ago
vLLM Narrows Gap with DeepInfer to 6% on NVIDIA GB200, Reaching 1M Tokens/s per GPU on GB300
Hours after DeepSeek open-sourced DeepInfer, the vLLM project team released official benchmark parity data serving DeepSeek-V4.1-Flash. Utilizing DeepSeek's identical decoding setup (131K context window, nextn=3 speculative decoding), vLLM closed its throughput performance delta to within 6% of DeepInfer across batch sizes 8 through 256 on NVIDIA GB200 systems. On GB300 hardware, vLLM attained up to 1,000,000 tokens/second per GPU and 826 tokens/second per user, generating $38.96 in hourly GPU revenue at DeepSeek off-peak rates. vLLM simultaneously shipped day-0 runtime support for FastH3-Omni and Spark-X2.6-4B.
Tooling19d ago
Ollama Cloud Launches US/EU Zero-Data-Retention Hosting for DeepSeek-V4.1-Flash-1017 with Claude Code CLI, While SemiAnalysis Reveals GLM-5.3 Was Built 100% via Post-Training
Ollama launched US- and EU-hosted Ollama Cloud with strict zero data retention and zero training, adding immediate support for DeepSeek-V4.1-Flash-1017 via one-line launch commands for Claude Code and OpenCode alongside Ollama v0.40.6. Meanwhile, SemiAnalysis revealed that GLM-5.3 was built entirely by post-training the previous GLM-5.2 base model without pretraining from scratch, achieving a 'night and day difference' in cybersecurity and agentic capabilities.
Infrastructure19d ago
vLLM and OpenRouter Launch Day-0 Serving for DeepSeek-V4.1-Flash-1017: 830K tok/s per GPU and 1.3B Tokens/$ TCO on SemiAnalysis AgentX
Following the release of DeepSeek-V4.1-Flash-1017, vLLM announced Day-0 serving optimization delivering 1.3B tokens per dollar TCO and 830K tok/s per GPU at 100 tok/s/user on SemiAnalysis AgentX (5x agentic throughput gain since Day 0), while OpenRouter launched live multi-provider routing starting at $0.10/M input and $0.50/M output tokens.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.