On 2026-11-05 Hyperstition open-sourced Hyperkernel’s first generated megakernels (Hyperserve), with vendor-reported gains versus NVIDIA-published vLLM baselines; Alpha, B300-oriented.
Zoom
Key Takeaways
Official: @hyperstition_cc on 2026-11-05; GitHub Hyperstitioncc/Hyperkernel (MIT, Alpha)
Vendor claim: DeepSeek v4.1 Flash ~36.7% higher per-user gen at C1 vs NVIDIA-published vLLM; ~90.6% at C4 vs AgentX
π₀-FAST ~94 vs ~243 ms; Matrix-Game ~37.4 vs ~11.6 FPS (~3.2×)
a16z's Charts of the Week for Oct 30 (by Moses Sternstein) looks at the "Rack Spread": GPU rental prices are rising while token prices fall. The official a16z X account summed it up on Oct 31: since May, average token prices are down 67% and token spend is up more than 120%. By the article's napkin math, Ramp data gives roughly 122%, and Ornn's blended price multiplied by OpenRouter volume gives roughly 160%. A separate post says that since mid-July B300 rentals are up 106%, B200 up 66% and H100 up 17%. a16z also stresses that these indexes come from small, biased samples and do not show true total demand.
Zoom
Key Takeaways
Prices and spend: since May, average token prices are down 67% and token spend is up more than 120% (a16z post on X, a16z figures)
Two napkin estimates: Ramp token-spend data shows about +122%; Ornn blended price times OpenRouter volume shows about +160% (Charts of the Week, described by the author as napkin math, not a rigorous method)
GPU rentals since mid-July: B300 +106%, B200 +66%, H100 +17%, while average token prices fell 58% (a16z post)
AI spend per employee: an Atlanta Fed survey puts the mean above $1,000, while Ramp's median is $15.70 a month ($188 a year)
Reach: the two posts have about 371 and 492 likes and 38K and 96K views (as of 18:15 SGT)
On Oct 28 DiffuSpace, whose members come from the HKU NLP group behind Dream 7B, released its first model, Clover-35B-A3B-Preview: a mixture-of-experts diffusion language model initialized from Qwen3.6-35B-A3B, with 35B total and about 3B active parameters, a 128K context window and block-parallel denoising. Vendor-reported results include 73.6 on SWE-bench Verified, 51.7 on Terminal-Bench 2.1 and 83.8 on GPQA Diamond, with 680 tokens/s on a single GB300 in agent workloads. Weights are Apache 2.0 on Hugging Face, ModelScope and GitHub, with SGLang and llama.cpp support.
Zoom
Key Takeaways
A 35B-total, roughly 3B-active MoE diffusion model initialized from Qwen3.6-35B-A3B, with 128K context and a thinking mode
DiffuSpace says it tops all six agent benchmarks it compared: SWE-bench Verified 73.6, Terminal-Bench 2.1 51.7 and tau3 Overall 65.3, ahead of LLaDA2.2 flash, Mercury 2.5 and Qwen3.6-35B-A3B
Speed (vendor-reported): 680 tokens/s on one GB300 at BF16 and batch size 1 in agent workloads, and 515 tokens/s on one H200 versus 365 tokens/s for Qwen with DFlash2
An INT4 build runs on a 16GB RTX 5070 Ti at 101.8 tokens/s on MATH-500
Weights on Hugging Face are Apache 2.0, about 69GB in BF16, and SGLang is the recommended serving path
Hours after DeepSeek open-sourced DeepInfer, the vLLM project team released official benchmark parity data serving DeepSeek-V4.1-Flash. Utilizing DeepSeek's identical decoding setup (131K context window, nextn=3 speculative decoding), vLLM closed its throughput performance delta to within 6% of DeepInfer across batch sizes 8 through 256 on NVIDIA GB200 systems. On GB300 hardware, vLLM attained up to 1,000,000 tokens/second per GPU and 826 tokens/second per user, generating $38.96 in hourly GPU revenue at DeepSeek off-peak rates. vLLM simultaneously shipped day-0 runtime support for FastH3-Omni and Spark-X2.6-4B.
Zoom
Key Takeaways
vLLM published direct benchmark comparisons, demonstrating that its serving throughput on NVIDIA GB200 is within 6% of DeepInfer
Achieves unprecedented scaling on NVIDIA GB300: up to 1,000,000 tokens/second per GPU and 826 tokens/second per active user stream
Generates up to $38.96 per GPU-hour in revenue when mapped to DeepSeek's production off-peak API pricing schedule
Delivers immediate Day-0 support for the FastH3-Omni video model and the 1M-token Apache 2.0 model Spark-X2.6-4B
On Oct 20, inference company Eagle released EAGLE-4, the fourth generation of the EAGLE speculative-decoding family. One drafter resizes its speculative tree for the batch size, hardware and workload while keeping the target model's output distribution. Eagle reports 961.6 tokens/s for one request on Qwen3.8-27B with 4x B300. The code ships as eagle-harness v0.1.0 with plugins for SGLang, vLLM and an experimental TokenSpeed backend. Many reposts call it open source, but the official license is source-available, not OSI open source.
Zoom
Key Takeaways
Vendor figure: 961.6 tokens/s on Qwen3.8-27B with TokenSpeed, BF16, 4x B300, TP4, one request, T=0 (paper Figure 1(a))
Repo measurements: vLLM on 4x B300 goes from 241.6 to 722.8 tok/s at concurrency 1 (2.99x) and 1.70x at concurrency 8; SGLang gets 1.91x at concurrency 8 and 1.35x at 64
AMD numbers too: SGLang on one MI355X gets 2.07x at concurrency 8 and 1.56x at 32 (ROCm 7.2)
Design: sliding-window draft layer, low-rank candidate head and append-only draft cache; the harness calibrates verify latency offline and grows the tree while gain per cost stays high
License: Eagle 4 Source-Available; free for noncommercial use, commercial use only under $20M trailing 12-month revenue and outside competing offerings
On Oct 20, NVIDIA's GEAR lab (Jim Fan's team) open-sourced Flora, a model suite that turns human and robot manipulation video into timestamped action captions, process rewards, outcome rewards and rubric scores. It includes Flora-Captioner 27B and Flora-Rewarder 27B (both on Qwen3.5-27B) and Flora-TempSeg 9B (a LoRA on Qwen3.5-9B), with weights on Hugging Face under an NVIDIA non-commercial license. NVIDIA says it is the in-house data engine used to train GR00T, labeling 200,000 hours of video a day on 512 GB300s, matching GPT-6 Astra on annotation quality at roughly 125x lower cost.
Zoom
Key Takeaways
Three models: TempSeg 9B segments actions, Captioner 27B writes one instruction per segment, and Rewarder 27B outputs dense process rewards and an episode success score
Cost (NVIDIA estimate): about $0.72 per hour of source video on eight GB300s at $10 per GPU-hour, versus about $88.49 for GPT-6 Astra at maximum effort, a 122-125x gap
Quality (NVIDIA 50-episode internal eval): 57.1% for Flora versus 53.7% for GPT-6 Astra; failure recall on DenseRewardBench is 53.3% while frontier API models mostly score under 1%
Scale: NVIDIA says Flora labels 200,000 hours of video a day on 512 GB300s to train GR00T-2
License: NVIDIA OneWay Noncommercial, research only, with commercial and production use explicitly unsupported
Meta's engineering blog on October 19 gave the first detailed look at Prometheus, the New Albany, Ohio supercluster that trained the Muse family, including Muse Wonder. Phase one is complete with hundreds of thousands of GPUs, and the full build targets about 1 million. Muse Wonder training reached more than 85% goodput in pretraining and more than 90% in reinforcement learning, with a mean time between interruptions of about 47K GPU hours, since raised above 54K by firmware fixes. It runs GB200 NVL72 and GB300 NVL72 platforms, with Vera Rubin NVL72 arriving soon.
Zoom
Key Takeaways
Scale: phase one trained the Muse family; it has hundreds of thousands of GPUs today, targets about 1 million, and runs across buildings as a single machine.
Reliability: Muse Wonder hit 85%+ pretraining goodput and 90%+ for RL; MTBI was about 47K GPU hours and is now above 54K.
Interruptions: a mid-2025 peak of nearly 1,000 events per 8K GPUs per day was cut by 99% over a year of engineering.
Sparing: each NVL72 rack runs a 64-GPU slice with 8 in-rack spares, plus about 2% spare racks; a 32,768-GPU job needs 523 racks or 37,656 GPUs, about 13% extra.
Network: each building fabric links up to 20,736 GPUs, and the aggregation network spans 2 km to over 100 km.
The SGLang team released v0.5.22 featuring 950 PRs from 245 contributors: DeepSeek-V4.1 delivers up to 2.3x more throughput per GPU on NVIDIA H200 and 1.9x on AMD MI350X/MI355X via DSpark. The new InstantTensor loader cuts model loading time by 10x. B200 decodes Qwen3.8-Flash-Next 12% faster, while B300 speeds up GLM-5.3-Flash by 10% and yields 1.85x decode throughput for MiMo-V2.6-Pro. SGLang also adds native Apple Silicon PyTorch support and ComfyUI integration.
Zoom
Key Takeaways
SGLang v0.5.22 released with 950 PRs from 245 contributors across heterogeneous platforms
DeepSeek-V4.1 achieves up to 2.3x throughput per GPU on NVIDIA H200 and 1.9x on AMD MI350X/MI355X via DSpark
New InstantTensor loader speeds up large model checkpoint ingestion by 10x
Qwen3.8-Flash-Next decodes 12% faster on B200; MiMo-V2.6-Pro reaches 1.85x decode throughput on B300
PD disaggregation cuts p95 TTFT in half (-50%) on long prompts, alongside native Apple Silicon and ComfyUI diffusion support
SemiAnalysis revealed the first Google TPUv7 benchmark submission for the 2.8T Kimi K3 MoE model, led end-to-end by Inferact (the creators of vLLM) and Google using the TorchTPU vLLM stack. At a hardware rental cost of just $1.21/chip-hour (vs $2.31 for GB300 and $1.86 for GB200), TPUv7 delivers ~2.2M tokens per $1 TCO in the ultra-high interactivity regime (~390 tok/s/user), outpacing NVIDIA GB200 NVL72, B300, and AMD MI355X on cost-performance.
Zoom
Key Takeaways
Inferact (creators of vLLM) and Google submitted their first preliminary TPUv7 benchmark for the 2.8T Kimi K3 MoE model
TPUv7 chip rental sits at $1.21/hour, significantly below GB300 ($2.31), B300 ($2.26), and GB200 ($1.86)
Delivers ~2.2M tokens per $1 TCO at ~390 tok/s/user, beating NVIDIA GB200 NVL72, B300, and AMD MI355X in the ultra-high interactivity regime
Built with fused TorchTPU vLLM megakernels and the TPU-Sync KVCache offload library for production PD disaggregation
vLLM and NVIDIA unveiled a unified CuTe DSL decode megakernel (~12,000 LoC authored by NVIDIA CentML's Benjamin Chislett) for Moonshot AI's frontier Kimi K3 model on NVIDIA GB300 NVL72. By fusing attention, MoE, and cross-GPU communication, the optimization cuts per-layer latency from 130 µs (23 kernels) to 69 µs (13 kernels), delivering 842 tokens/s at batch=1 with DSpark speculative decoding (1.73x faster than stock vLLM) and up to 1.8x higher P90 per-user interactivity (403.2–423 tok/s/user) on SemiAnalysis AgentX, with DeepSeek-V4.1-Flash (PR #61868) and GLM-5.3 support coming next.
Zoom
Key Takeaways
NVIDIA and vLLM introduce a 12,000-line CuTe DSL decode megakernel for the 2.8T-parameter Kimi K3, reducing per-layer execution from 23 kernels (130 µs) to 13 kernels (69 µs)
On 16x NVIDIA GB300 NVL72 GPUs, batch-1 decode hits 842 tok/s with DSpark speculative decoding (1.73x faster) and 256.4 tok/s without speculation (1.88x faster)
Delivers up to 1.8x higher P90 interactivity (403.2-423 tok/s/user) on SemiAnalysis AgentX, with DeepSeek-V4.1-Flash (vLLM PR #61868) and GLM-5.3 megakernels following next
A vLLM blog post by NVIDIA engineer Benjamin Chislett details an agent-assisted CuTe DSL decode path for Kimi K3 on NVIDIA GB300 that fuses attention, MoE and NVLink communication, cutting per-layer time from 130 µs across 23 kernels to 69 µs across 13. With DSpark speculative decoding on 16 GPUs it reaches 842 tok/s at batch 1, 1.73x stock vLLM; PR #61867 was merged on Oct 16.
Zoom
Key Takeaways
842 tok/s at batch 1 on 16 GB300 GPUs with DSpark (7 draft tokens, synthetic AL=6) vs 488 for stock vLLM, 1.73x (vendor figures)
Without speculation TP16 gains 1.62–1.92x (3,606 tok/s at batch 64); TP8 gains 1.31–1.58x
Per layer: 23 kernels and 130 µs down to 13 kernels and 69 µs
AgentX P90 interactivity 403.2 vs 223.7 tok/s/user (1.8x), though the baseline is an earlier 8-GPU run
The same fast-track approach now covers DeepSeek-V4.1-Flash (PR #61868, open); early Muse Spark Open results exceed 950 tok/s/user
SemiAnalysis published the architectural breakdown behind the 1.3B tokens per dollar TCO record on AgentX: vLLM and Inferact utilized Prefill/Decode disaggregated serving at hyperscaler volume pricing to achieve 4x better cost-efficiency than an aggregated NVIDIA B300 configuration. Concurrently, open-source coding agent Cline reported that DeepSeek-V4.1-Flash-1017 beats GPT-5.6-Sol on Terminal-Bench at ~17x lower cost (+35% over prior V4.1-Flash) purely by scaling RL training across multiple coding agent harnesses.
Zoom
Key Takeaways
SemiAnalysis confirms vLLM and Inferact achieve 1.3B tokens per dollar TCO on AgentX via disaggregated serving—4x better than an aggregated NVIDIA B300 configuration
Cline reports that DeepSeek-V4.1-Flash-1017 improves prior V4.1-Flash Terminal-Bench performance by 35%, beating GPT-5.6-Sol at ~17x lower cost
The 35% agentic coding leap was achieved without base algorithmic changes by scaling reinforcement learning directly across multiple coding agent harnesses
Addressing KV cache bottlenecks in long-context LLM serving, a new production study on 8xB300 nodes serving GLM-5.3 (arXiv:2610.21500) reveals that unsharded remote KV caches inflate hit-heavy p99 TTFT to 83.32s, and resolves it via rank-sharded asynchronous LMCache loading—boosting GET bandwidth 6.3x to 10.76 GB/s and slashing p99 to 0.67s (+34% throughput). Meanwhile, Baidu PaddlePaddle open-sources Virtual Head Attention (VHA, arXiv:2610.17959), folding linear Q-Premix and Postmix into standard GQA-2 weights at inference to halve KV cache while beating MLA (+0.70%) and GQA-4 (+2.60%) on a 30B-A3B MoE and lifting 8K-128K RULER from 73.0 to 84.4.
Zoom
Key Takeaways
On 8xB300 nodes serving GLM-5.3 with PCP8, sharding LMCache across ranks and adding async loading boosts GET bandwidth 6.3x to 10.76 GB/s, cutting hit-heavy p99 TTFT from 83.32s to 0.67s (+34% throughput)
Baidu's Virtual Head Attention (VHA) folds linear Q-Premix and low-rank Postmix directly into W_Q and W_O at inference, reducing to a zero-overhead GQA-2 architecture with half to a quarter of the KV cache
On a 500B-token 30B-A3B MoE, VHA outperforms both MLA (+0.70%) and GQA-4 (+2.60%), and boosts 8K-128K RULER on a 10B-A1B MoE from 73.0 (GQA-4) to 84.4 using only 2 KV heads
On Oct 9 the vLLM team, with Inferact, Red Hat and NVIDIA, shipped early Vera Rubin NVL72 support: CUDA 13.4 nightly images already serve DeepSeek, Kimi, GLM and MiniMax. Vendor-reported results: up to 7.84x per-GPU throughput vs GB200 on SemiAnalysis AgentX (MiniMax M3, matched interactivity) and up to 3.7x vs GB300 NVL72 on the MLPerf Inference v6.1 VLM benchmark (Qwen3-VL-235B-A22B).
On 2026-10-01 Ai2 released Olmo-core 3, an open MoE training stack (GitHub, PyPI ai2-olmo-core). It moves from FSDP to DDP + expert parallelism; on 8×B300 a 47B MoE hits ~2.7× throughput (52k vs 19.4k tok/s/GPU). Systems benches include 1.2T on 512 GPUs at up to 858 useful-model TFLOP/s/GPU and a DeepEP v2 capacity probe at 2.38T. Report: olmocore3.
Zoom
Key Takeaways
Announced 2026-10-01: allenai.org/blog/olmocore3 + HF mirror + github.com/allenai/OLMo-core
