On 2026-11-05, Hugging Face Harness Optimization researcher Joël Niklaus (@joelniklaus) released the Harness Leaderboard: same model, provider, and reasoning effort with only the harness swapped, covering GLM-5.3-Flash, DeepSeek-V4.1-Flash, and Flint on Terminal-Bench 2.1 and DeepSWE—28,849 trials and about $4,746 of inference. Author-reported averages: mini-swe-agent ~51.4% resolve rate; Pi ~49.4% at about $0.29 per solved task. This is an individual research leaderboard, not a Hugging Face company product announcement.
Zoom
Key Takeaways
Source: personal post by HF Harness Optimization researcher Joël Niklaus on 2026-11-05T16:04:08Z—not a company announcement
Scale (author): 9 harnesses on open models, 28,849 trials, ~$4,746 inference spend
Method: pin model/provider/reasoning effort; 3 attempts per task; Terminal-Bench 2.1 and DeepSWE
Results (author): averaged mini-swe-agent ~51.4%, openclaw ~27.3%; Pi ~49.4% at ~$0.29 per solved task
Use: public HF Space leaderboard + trajectories; Harness Lens documents prompts/tools for each harness
The vLLM project team and Ant Group officially open-sourced Humming (humming-kernels), a specialized high-performance kernel library designed for serving quantized MoE and dense LLMs at massive scale. Humming natively supports WnA16, W8A8, W4A8, and W4A4 across INT1–INT8 and FP4–FP8 formats, spanning NVIDIA architectures from Turing through Blackwell as well as T-Head Zhenwu 810E silicon. In production benchmarks, Humming kernels outpace cuBLAS by 3–4x at small batch sizes and surpass Marlin by 2x on H200 clusters. Serving DeepSeek-V4.1-Flash across 8x H20 GPUs, Humming drives a +43.3% throughput gain under 64 concurrent requests while reducing time-to-first-token (TTFT) by 32%, pre-integrated directly into vLLM v0.29.0+.
Zoom
Key Takeaways
vLLM and Ant Group open-sourced Humming (humming-kernels), integrated out-of-the-box into vLLM v0.29.0+
Covers WnA16, W8A8, W4A8, and W4A4, compiling JIT kernels in just 1.82 seconds versus 23.35 seconds for CUTLASS
Serving DeepSeek-V4.1-Flash on 8x H20 GPUs yields a +43.3% throughput boost at 64 concurrency (442 to 634 tokens/sec)
Cuts time-to-first-token (TTFT) by 25% to 32% while delivering up to 5x kernel speedups on Blackwell B300 systems
On Oct 21 the vLLM community and Ant Group open-sourced Humming (Apache 2.0), a quantized-inference library that JIT-compiles GEMM kernels on demand. It supports every NVIDIA architecture from Turing on plus T-Head's Zhenwu 810E, and covers weight and activation combinations from INT1-INT8 and FP4-FP8 for dense and MoE workloads. In vLLM's microbenchmarks it runs 2-5x faster than vLLM Marlin, FlashInfer or DeepGEMM in some cases; on 8 H20 GPUs, switching DeepSeek-V4.1-Flash MoE from Marlin W4A16 to Humming W4A8 raised throughput 14-43%. vLLM 0.29.0 ships it built in.
Zoom
Key Takeaways
One JIT kernel framework instead of a hand-written kernel per format: WNA16, W8A8, W4A8, W4A4, INT1-INT8 and FP4-FP8 weights, dense and MoE
Hardware: every NVIDIA GPU from Turing (SM75) on, including Blackwell SM100/103/120/121, plus T-Head Zhenwu 810E
End to end: DeepSeek-V4.1-Flash on 8x H20 at TP=8, Humming W4A8 beats Marlin W4A16 by 14-43% in throughput and cuts mean TTFT 25-32%; at concurrency 64 output goes from 442 to 634 tok/s
Microbenchmarks run by the vLLM community: W4A8-MXFP4 on H200 is about 4.5x DeepGEMM W8A8, and the B300 MoE Grouped path is about 5x Marlin; one W4A16 kernel compiles in 1.82 s versus 23.35 s for CUTLASS
Use: pip install humming-kernels (about a 500 KB wheel); built into vLLM 0.29.0+, enable with --linear-backend humming --moe-backend humming
Hours after DeepSeek open-sourced DeepInfer, the vLLM project team released official benchmark parity data serving DeepSeek-V4.1-Flash. Utilizing DeepSeek's identical decoding setup (131K context window, nextn=3 speculative decoding), vLLM closed its throughput performance delta to within 6% of DeepInfer across batch sizes 8 through 256 on NVIDIA GB200 systems. On GB300 hardware, vLLM attained up to 1,000,000 tokens/second per GPU and 826 tokens/second per user, generating $38.96 in hourly GPU revenue at DeepSeek off-peak rates. vLLM simultaneously shipped day-0 runtime support for FastH3-Omni and Spark-X2.6-4B.
Zoom
Key Takeaways
vLLM published direct benchmark comparisons, demonstrating that its serving throughput on NVIDIA GB200 is within 6% of DeepInfer
Achieves unprecedented scaling on NVIDIA GB300: up to 1,000,000 tokens/second per GPU and 826 tokens/second per active user stream
Generates up to $38.96 per GPU-hour in revenue when mapped to DeepSeek's production off-peak API pricing schedule
Delivers immediate Day-0 support for the FastH3-Omni video model and the 1M-token Apache 2.0 model Spark-X2.6-4B
DeepSeek officially open-sourced DeepInfer under Apache 2.0, its production-grade inference engine serving DeepSeek-V4.1-Flash at scale. Originating from vLLM with architectural refinements from SGLang and TensorRT-LLM, DeepInfer natively targets NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It integrates specialized kernels including FlashMLA, DeepGEMM, DeepEP, and TileKernels. The engine features DSpark confidence-scheduled speculative decoding with 5-token drafts, SWA bounded replay over a 140-token sliding window, and balanced MoE expert parallelism. DeepSeek simultaneously open-sourced deepseek-recipe, delivering an end-to-end open ecosystem across models, kernels, and inference runtimes.
Zoom
Key Takeaways
DeepSeek officially open-sourced DeepInfer under Apache 2.0, releasing the production codebase powering DeepSeek-V4.1-Flash at massive scale
Natively optimized for both NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs with deterministic, batch-independent output guarantees
Integrates custom FlashMLA, DeepGEMM, DeepEP, and TileKernels, alongside DSpark 5-token speculative decoding and 140-token SWA replay
Simultaneously released deepseek-recipe, aligning open-source local deployments perfectly with DeepSeek's official production API tokenization
On Oct 20 (UTC) DeepSeek open-sourced DeepInfer (Apache 2.0), the in-house inference stack that serves DeepSeek-V4.1-Flash at scale. The release covers offline inference with Attention DP/CP and MoE EP+EPLB parallelism, DSpark speculative decoding, SWA bounded replay, batch-invariant inference, deterministic sampling and structured outputs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The KV cache system, prefill/decode disaggregation and load balancing are not included. DeepSeek also released deepseek-recipe (MIT), which converts Chat Completions, Messages and Responses requests and encodes them the same way the official API does.
Zoom
Key Takeaways
DeepInfer is Apache 2.0 and, per DeepSeek, is the framework serving DeepSeek-V4.1-Flash in production; it currently supports only that model
Hardware is limited to NVIDIA SM100 (Blackwell) GPUs and Huawei Ascend 950 NPUs; the official Pareto curves were measured on 4 GPUs for prefill and 64 for decode
The open release omits the KV cache system, VE/prefill/decode disaggregation and cross-replica load balancing, so it is not a full serving stack yet
DeepSeek's reference V4.1 scores (Pass@1, vendor-reported): AIME 2026 99.9%, GPQA Diamond 93.4%, HLE-Full 42.6%, meant as a deployment sanity check
deepseek-recipe (MIT, Rust with Python bindings) handles Chat Completions/Messages/Responses conversion and token encoding/decoding
On Oct 19 (UTC) Unsloth released dynamic GGUF quants and a run-locally guide for DeepSeek-V4.1-Flash-1017. Per Unsloth, the 1-bit UD-IQ1_M quant is 250.4 GB (vs 508.4 GB for lossless Q8), keeps 88.2% top-1 agreement with the official weights on long17k, and runs on 128 GB+ RAM machines, because the model's 110.6–202.8 GB Engram n-gram table can stay on SSD via mmap. Running it currently requires Unsloth's llama.cpp PR build or Unsloth Desktop.
Zoom
Key Takeaways
1-bit UD-IQ1_M is 250.4 GB, about 51% smaller than lossless UD-Q8_K_XL (508.4 GB), with 88.2% top-1 agreement on long17k and 87.1% on wiki1100 (Unsloth-reported)
The smallest UD-IQ1_S (233.0 GB) targets 128 GB RAM machines such as a Mac or DGX Spark at 82.8–85.1% agreement; 3-bit UD-IQ3_XXS needs ~256 GB (e.g. Mac Studio) at 94.6–95.1%
The model pairs a 552B backbone with 196B Engram parameters; the 110.6–202.8 GB Engram n-gram table can stay on SSD via mmap, saving RAM but not disk
Unsloth claims ~3x faster inference than other CPU/VRAM/ngram-focused repos on top of llama.cpp (vendor claim, no independent test yet)
The Hugging Face repo is MIT-licensed with 9 quant levels from IQ1_S to Q8_K_XL plus standalone Engram files at different precisions
Researchers introduced BlenderWorld (arXiv:2610.23986), the first comprehensive benchmark evaluating visual agents creating and editing 3D models from real-world references in Blender. Spanning 223 tasks across procedural geometry, Python scripting, material shaders, spatial layouts, and lighting, GPT-6 Astra leads the benchmark with a composite Score of 79.9 (and a 42.2% Perfect completion rate), followed by Opus-5 at 73.0. Open-weight flagship DeepSeek-V4.1-Flash claims 65.7, matching proprietary models while relying on active execution-and-inspection tool loops.
Zoom
Key Takeaways
Researchers introduced BlenderWorld (arXiv:2610.23986), a 223-task benchmark evaluating autonomous visual agents creating 3D content in Blender
GPT-6-Astra leads the world with a 79.9 composite Score and 42.2% Perfect completion rate, followed by Opus-5 at 73.0
DeepSeek-V4.1-Flash ranks #1 among open models with 65.7, matching proprietary GPT-5.6-Sol on multimodal 3D synthesis (65.5)
The SGLang team released v0.5.22 featuring 950 PRs from 245 contributors: DeepSeek-V4.1 delivers up to 2.3x more throughput per GPU on NVIDIA H200 and 1.9x on AMD MI350X/MI355X via DSpark. The new InstantTensor loader cuts model loading time by 10x. B200 decodes Qwen3.8-Flash-Next 12% faster, while B300 speeds up GLM-5.3-Flash by 10% and yields 1.85x decode throughput for MiMo-V2.6-Pro. SGLang also adds native Apple Silicon PyTorch support and ComfyUI integration.
Zoom
Key Takeaways
SGLang v0.5.22 released with 950 PRs from 245 contributors across heterogeneous platforms
DeepSeek-V4.1 achieves up to 2.3x throughput per GPU on NVIDIA H200 and 1.9x on AMD MI350X/MI355X via DSpark
New InstantTensor loader speeds up large model checkpoint ingestion by 10x
Qwen3.8-Flash-Next decodes 12% faster on B200; MiMo-V2.6-Pro reaches 1.85x decode throughput on B300
PD disaggregation cuts p95 TTFT in half (-50%) on long prompts, alongside native Apple Silicon and ComfyUI diffusion support
Arena announced a major update to Code Arena: WebDev: DeepSeek-V4.1-Flash-20261017 (Max) clinched the #1 position among open models with 1670 points, surpassing Kimi K3 (Max) at a 10x lower cost. Ranking #12 overall, it outperforms Claude Opus 5 (High) and narrows the gap with closed-source frontier GPT-6 Sol (Max) to roughly 10 points, resetting the Pareto efficiency curve at $0.30 input / $1.20 output per million tokens.
Zoom
Key Takeaways
DeepSeek-V4.1-Flash-20261017 (Max) captured the #1 open model position in Arena Code Arena: WebDev with 1670 points, surpassing Kimi K3 (Max)
Ranks #12 overall on the leaderboard, defeating Claude Opus 5 (High) and trailing closed-source champion GPT-6 Sol (Max) by only ~10 points
Resets the Pareto efficiency frontier at $0.30 input / $1.20 output per million tokens, cutting operational costs by ~10x against rival models
Gains +52 points over its previous 20260910 checkpoint in just over a month
vLLM and NVIDIA unveiled a unified CuTe DSL decode megakernel (~12,000 LoC authored by NVIDIA CentML's Benjamin Chislett) for Moonshot AI's frontier Kimi K3 model on NVIDIA GB300 NVL72. By fusing attention, MoE, and cross-GPU communication, the optimization cuts per-layer latency from 130 µs (23 kernels) to 69 µs (13 kernels), delivering 842 tokens/s at batch=1 with DSpark speculative decoding (1.73x faster than stock vLLM) and up to 1.8x higher P90 per-user interactivity (403.2–423 tok/s/user) on SemiAnalysis AgentX, with DeepSeek-V4.1-Flash (PR #61868) and GLM-5.3 support coming next.
Zoom
Key Takeaways
NVIDIA and vLLM introduce a 12,000-line CuTe DSL decode megakernel for the 2.8T-parameter Kimi K3, reducing per-layer execution from 23 kernels (130 µs) to 13 kernels (69 µs)
On 16x NVIDIA GB300 NVL72 GPUs, batch-1 decode hits 842 tok/s with DSpark speculative decoding (1.73x faster) and 256.4 tok/s without speculation (1.88x faster)
Delivers up to 1.8x higher P90 interactivity (403.2-423 tok/s/user) on SemiAnalysis AgentX, with DeepSeek-V4.1-Flash (vLLM PR #61868) and GLM-5.3 megakernels following next
Ollama launched US- and EU-hosted Ollama Cloud with strict zero data retention and zero training, adding immediate support for DeepSeek-V4.1-Flash-1017 via one-line launch commands for Claude Code and OpenCode alongside Ollama v0.40.6. Meanwhile, SemiAnalysis revealed that GLM-5.3 was built entirely by post-training the previous GLM-5.2 base model without pretraining from scratch, achieving a 'night and day difference' in cybersecurity and agentic capabilities.
Zoom
Key Takeaways
Ollama Cloud launches US- and EU-hosted zero-data-retention serving for DeepSeek-V4.1-Flash-1017 with one-line ollama launch integrations for Claude Code and OpenCode
Ollama v0.40.6 adds GGUF q2_0 tensor support, clef cloud integration, and fixes orphaned runner processes on Windows
SemiAnalysis reveals GLM-5.3 was built entirely by post-training the previous model without pretraining from scratch, delivering a 'night and day difference' in cyber capabilities over GLM-5.2
SemiAnalysis published the architectural breakdown behind the 1.3B tokens per dollar TCO record on AgentX: vLLM and Inferact utilized Prefill/Decode disaggregated serving at hyperscaler volume pricing to achieve 4x better cost-efficiency than an aggregated NVIDIA B300 configuration. Concurrently, open-source coding agent Cline reported that DeepSeek-V4.1-Flash-1017 beats GPT-5.6-Sol on Terminal-Bench at ~17x lower cost (+35% over prior V4.1-Flash) purely by scaling RL training across multiple coding agent harnesses.
Zoom
Key Takeaways
SemiAnalysis confirms vLLM and Inferact achieve 1.3B tokens per dollar TCO on AgentX via disaggregated serving—4x better than an aggregated NVIDIA B300 configuration
Cline reports that DeepSeek-V4.1-Flash-1017 improves prior V4.1-Flash Terminal-Bench performance by 35%, beating GPT-5.6-Sol at ~17x lower cost
The 35% agentic coding leap was achieved without base algorithmic changes by scaling reinforcement learning directly across multiple coding agent harnesses
Following the release of DeepSeek-V4.1-Flash-1017, vLLM announced Day-0 serving optimization delivering 1.3B tokens per dollar TCO and 830K tok/s per GPU at 100 tok/s/user on SemiAnalysis AgentX (5x agentic throughput gain since Day 0), while OpenRouter launched live multi-provider routing starting at $0.10/M input and $0.50/M output tokens.
Zoom
Key Takeaways
vLLM delivers 830K tok/s per GPU (cached + input + output) at 100 tok/s per user and 1.3B tokens/$ TCO on SemiAnalysis AgentX for DeepSeek-V4.1-Flash-1017 (5x faster since Day 0)
DeepSeek-V4.1-Flash-1017 reaches 77.0% on DeepSWE v1.1 (beating Claude Opus 5 at 74.0%), 68.2% on NL2Repo-Bench, and 72.1% on Apex Bench with shorter agentic outputs
Live on OpenRouter from $0.10/M input and $0.50/M output tokens, while DeepSeek's official API alias deepseek-flash remains at $0.15/M input and $0.60/M output
On 2026-10-15 Microsoft AI published a technical post and PDF saying it has finished pre-training its in-house base model MAI-Base-2: MoE in every layer (1 shared + 7-of-512 routed experts), 80B active / 2.8T total, 102 layers, native encoder-free vision, a 50T-token mixture and a March 2026 knowledge cutoff. Per MAI, it beats every open-weight model tested on held-out BPB and its recipe is 4.2–12.3x more pre-training-FLOPs-efficient than DeepSeek-V4-Pro-Base. It is still in post-training: no API, no weights and no release date yet; Microsoft plans to license weights to trusted partners via Foundry.
Zoom
Key Takeaways
Scale: 80B active / 2.8T total, 102 layers, hidden 8704, MoE in every layer (1 shared + 7 of 512 routed); MAI-Base-1 was 35B/1T, 78 layers, alternating dense/MoE (PDF)
Compute: 16K GB200s for 38 days (Aug–Sep 2026), 21% average MFU (vs 2.5 PFLOPS BF16), 89% goodput; the Muon step is 2.1% (165 ms) of iteration time
Efficiency (vendor estimate via scaling-law fits): 4.2–12.3x fewer FLOPs than DeepSeek-V4-Pro-Base to match its BPB; 6.3–12.7x better than the June MAI-Base-1 recipe
Data: 50T-token mixture, 18% vision-language (4.5T unique), ~36% more unique tokens than Base-1, described as organic data with no distillation
Availability: still in post-training, no API/weights/date; weights to be licensed to trusted partners through Foundry; MAI-Base-3 prep underway (blog)
