When Remote KV Cache Is Slower Than GPU Recompute: Sharded LMCache Cuts p99 from 83.32s to 0.67s on 8xB300, While Baidu VHA Halves KV Cache and Beats MLA
Addressing KV cache bottlenecks in long-context LLM serving, a new production study on 8xB300 nodes serving GLM-5.3 (arXiv:2610.21500) reveals that unsharded remote KV caches inflate hit-heavy p99 TTFT to 83.32s, and resolves it via rank-sharded asynchronous LMCache loading—boosting GET bandwidth 6.3x to 10.76 GB/s and slashing p99 to 0.67s (+34% throughput). Meanwhile, Baidu PaddlePaddle open-sources Virtual Head Attention (VHA, arXiv:2610.17959), folding linear Q-Premix and Postmix into standard GQA-2 weights at inference to halve KV cache while beating MLA (+0.70%) and GQA-4 (+2.60%) on a 30B-A3B MoE and lifting 8K-128K RULER from 73.0 to 84.4.
Key Takeaways
- On 8xB300 nodes serving GLM-5.3 with PCP8, sharding LMCache across ranks and adding async loading boosts GET bandwidth 6.3x to 10.76 GB/s, cutting hit-heavy p99 TTFT from 83.32s to 0.67s (+34% throughput)
- Baidu's Virtual Head Attention (VHA) folds linear Q-Premix and low-rank Postmix directly into W_Q and W_O at inference, reducing to a zero-overhead GQA-2 architecture with half to a quarter of the KV cache
- On a 500B-token 30B-A3B MoE, VHA outperforms both MLA (+0.70%) and GQA-4 (+2.60%), and boosts 8K-128K RULER on a 10B-A1B MoE from 73.0 (GQA-4) to 84.4 using only 2 KV heads
Linked Models & Agents:
