[{"data":1,"prerenderedAt":50},["Reactive",2],{"news-story-news-1792368356004-ef0uo":3},{"id":4,"title_zh":5,"title_en":6,"summary_zh":7,"summary_en":8,"takeaways_zh":9,"takeaways_en":14,"content_zh":19,"content_en":20,"external_links":21,"source":28,"category":35,"impact_score":36,"views_count":37,"likes_count":38,"related_entities":39,"tags":43,"image_url":49},"news-1792368356004-ef0uo","SemiAnalysis 披露 Google 与 Inferact 首份 TPUv7 评测：TorchTPU vLLM 大核驱动 2.8T Kimi K3，高交互吞吐力压 GB200 NVL72 与 B300","SemiAnalysis Discloses Google & Inferact TPUv7 Submission: TorchTPU vLLM Megakernels Drive 2.8T Kimi K3, Surpassing GB200 NVL72 and B300 in High-Interactivity TCO","半导体分析机构 SemiAnalysis 发布最新评测（X: 2111938013852762351）：Inferact（vLLM 核心团队）联合 Google 正式提交了首份基于 Google TPUv7 的 Kimi K3 2.8T 巨型 MoE 模型推理基准。在单芯片时租仅 1.21 美元（相比 GB300 2.31 美元、B300 2.26 美元、GB200 1.86 美元）下，借助 TorchTPU vLLM 大核与 TPU-Sync KVCache 卸载库，TPUv7 在 390 tok/s 超高交互区实现每美元 220 万 Token 吞吐，TCO 表现越级反超 NVIDIA GB200 NVL72、B300 与 AMD MI355X。","SemiAnalysis revealed the first Google TPUv7 benchmark submission for the 2.8T Kimi K3 MoE model, led end-to-end by Inferact (the creators of vLLM) and Google using the TorchTPU vLLM stack. At a hardware rental cost of just $1.21/chip-hour (vs $2.31 for GB300 and $1.86 for GB200), TPUv7 delivers ~2.2M tokens per $1 TCO in the ultra-high interactivity regime (~390 tok/s/user), outpacing NVIDIA GB200 NVL72, B300, and AMD MI355X on cost-performance.",[10,11,12,13],"Inferact（vLLM 核心团队）联合 Google 提交首份 Google TPUv7 Kimi K3 2.8T 巨型 MoE 推理基准","TPUv7 单芯片时租仅 1.21 美元，显著低于 GB300（2.31 美元）、B300（2.26 美元）与 GB200（1.86 美元）","在 390 tok/s 超高交互区实现每美元 220 万 Token 吞吐，TCO 表现越级反超 NVIDIA GB200 NVL72、B300 与 AMD MI355X","基于 TorchTPU vLLM 大核与 TPU-Sync KVCache 库，标志着 vLLM 在 Google TPU 架构上取得生产级重大突破",[15,16,17,18],"Inferact (creators of vLLM) and Google submitted their first preliminary TPUv7 benchmark for the 2.8T Kimi K3 MoE model","TPUv7 chip rental sits at $1.21/hour, significantly below GB300 ($2.31), B300 ($2.26), and GB200 ($1.86)","Delivers ~2.2M tokens per $1 TCO at ~390 tok/s/user, beating NVIDIA GB200 NVL72, B300, and AMD MI355X in the ultra-high interactivity regime","Built with fused TorchTPU vLLM megakernels and the TPU-Sync KVCache offload library for production PD disaggregation","### 核心背景与行业痛点\n随着 2.8T 规模参数的超巨型混合专家（MoE）智能体模型（如 Moonshot Kimi K3）成为生产级 Agent 推理主力，算力集群的推理总拥有成本（TCO）与单用户并发交互速度（Tokens/s per user）构成了核心矛盾。在实时交互式编码智能体中，开发者需要高达 200~400 tok/s 的流式响应以避免思维打断，而现有英伟达 B200/GB200 NVL72 集群在极高并发交互区由于通信开销与显存带宽饱和，每美元产出的 Token 量剧烈衰减。SemiAnalysis 独家披露了 **Google 与 Inferact（vLLM 商业化团队）** 联手打造的 **首份 Google TPUv7 官方推理性能提报（InferenceX Preview）**。\n\n### 架构亮点与底层机制\n1. **硬件成本端显著优势（Chip-Hour Cost）**：\n   - 根据 SemiAnalysis 披露的大型云厂商规模采购 TCO 测算，**Google TPUv7 单芯片时租仅为 $1.21 / 小时**，相比 AMD MI355X（$1.50）、NVIDIA B200（$1.73）、GB200（$1.86）、B300（$2.26）以及顶级 GB300（$2.31），具备压倒性的单位硬件成本优势；\n2. **TorchTPU vLLM 巨核与 TPU-Sync KVCache 库协同**：\n   - 由 vLLM 核心作者 Jeff Ma、George Novack 牵头，采用 **TorchTPU vLLM 深度融合架构**，针对 TPUv7 的矩阵乘法单元（MXU）编写了端到端 Fused Megakernel，并引入 `TPU-Sync KVCache` 跨节点显存传输与分离式预填充/解码（PD Disaggregation）卸载机制，彻底打通了 PyTorch/vLLM 生态在 Google 自研 TPU 上的高性能落地通路。\n\n### 权威 Benchmark 与实测跑分对比\n在 SemiAnalysis 发布的 **“Kimi K3 2.8T Agentic Total Tokens per $1 TCO vs. P90 Interactivity”** 权威曲线中：\n- **超高交互区间性能反超（~390 tok/s per user）**：\n  - 在每秒 390 Token 的高交互极限工况下，TPUv7（vLLM）产出达 **约 2.2M Tokens / $1 TCO**，全面超越 NVIDIA GB200 NVL72、B300、B200 以及 AMD MI355X；\n  - AMD MI355X 交互极限在 130 tok/s 处饱和截断，NVIDIA B300 在 270 tok/s 处饱和截断，而 TPUv7 可顺畅支撑高达 400 tok/s 的单用户并发流式输出；\n- **GB300 NVL72 对标态势**：\n  - 虽然 GB300 NVL72 在低交互（25 tok/s）工况下以 31M Tokens/$1 领先，但在 400 tok/s 极高交互区（~2.5M Tokens/$1），TPUv7 凭借仅为其一半的单芯片成本（$1.21 vs $2.31）实现了极具竞争力的平替性能；SemiAnalysis 指出在后续结合更优 PD 分离比后，外部客户成本维度亦有望对标 GB300。\n\n### 开发者实战落地与开箱指南\n- **开源生态意义**：该评测标志着 **vLLM 已成为跨越 NVIDIA（CUDA）、AMD（ROCm）与 Google（TPU）三大异构计算体系的通用统一服务层**；\n- **多云多架构部署建议**：对于自建大规模 Kimi K3 2.8T 或开源巨型 MoE 服务的企业，无需再被单一硬件厂商绑定。通过采用最新的 `TorchTPU` + `vLLM` 镜像，开发者即可在 Google Cloud TPUv7 Pods 上无缝迁移运行大模型推理集群，并在高交互 Agent 场景下取得极为可观的 TCO 降本回报。","### 核心背景与行业痛点\nAs massive ~2.8T-parameter Mixture-of-Experts (MoE) agent models like Moonshot Kimi K3 enter production, serving costs and user interactivity (tokens/s per user) become critical battlegrounds. For interactive coding agents, developers demand streaming latencies exceeding 200–400 tok/s to enable real-time flow states. However, traditional GPU clusters suffer steep TCO degradation at ultra-high interactivity due to communication overhead and memory bandwidth saturation. SemiAnalysis released an exclusive analysis covering **Google and Inferact's (makers of vLLM) first preliminary TPUv7 submission** on the InferenceX platform.\n\n### 架构亮点与底层机制\n1. **Hardware Cost Advantage ($1.21/chip-hour)**:\n   - Based on large-hyperscaler volume TCO calculations, **Google TPUv7 costs just $1.21 per chip-hour**, compared to AMD MI355X ($1.50), NVIDIA B200 ($1.73), GB200 ($1.86), B300 ($2.26), and GB300 ($2.31);\n2. **TorchTPU vLLM Stack & Megakernel Execution**:\n   - Led end-to-end by Inferact's Jeff Ma, George Novack, and Google, the stack leverages fused megakernels customized for TPUv7 MXUs alongside the `TPU-Sync KVCache` transfer and offloading library for prefill-decode (PD) disaggregation, demonstrating first-class production readiness for the PyTorch/vLLM stack on Google silicon.\n\n### 权威 Benchmark 与实测跑分对比\nFrom the **Kimi K3 2.8T Agentic Total Tokens per $1 TCO vs. P90 Interactivity** benchmark curves:\n- **Ultra-High Interactivity Regime (~390 tok/s per user)**:\n  - TPUv7 achieves **~2.2M tokens per $1 TCO**, outperforming NVIDIA GB200 NVL72, B200, B300, and AMD MI355X;\n  - AMD MI355X truncates around 130 tok/s and NVIDIA B300 around 270 tok/s, whereas TPUv7 smoothly sustains interactivity up to ~400 tok/s;\n- **Comparison with GB300 NVL72**:\n  - While GB300 leads at low interactivity (31M tokens/$1 at 25 tok/s), at ~400 tok/s (~2.5M tokens/$1 for GB300), TPUv7's lower cost per chip-hour ($1.21 vs $2.31) renders it highly competitive on a TCO basis.\n\n### 开发者实战落地与开箱指南\n- **Ecosystem Shift**: Proves that vLLM has matured into a truly portable runtime across NVIDIA (CUDA), AMD (ROCm), and Google (TPU) hardware;\n- **Deployment Strategy**: Enterprise clusters serving massive MoE models can leverage the open TorchTPU vLLM stack to exploit Google Cloud TPUv7 pods, unlocking compelling TCO efficiencies for interactive coding agents.",[22,25],{"title":23,"url":23,"type":24},"https://inferencex.semianalysis.com/inference/kimi-k3","other",{"title":26,"url":26,"type":27},"https://github.com/vllm-project/vllm","github",{"platform":29,"author_name":30,"author_handle":31,"author_avatar":32,"tweet_url":33,"published_at":34},"twitter","SemiAnalysis","@SemiAnalysis_","https://unavatar.io/x/SemiAnalysis_","https://x.com/SemiAnalysis_/status/2111938013852762351","2026-10-19T00:05:56.004Z","Hardware",93,54,0,{"models":40,"agents":41,"vendors":42},[],[],[],[30,44,45,46,47,48],"TPUv7","vLLM","Kimi K3","InferenceX","Google Cloud","https://pbs.twimg.com/media/HU8cbnUWsAAPOBg.jpg",1792905127218]