CMU Infini-AI Lab Drops CompressionBench and ServerBench: Coding Agents Compress LLMs and Design Serving Stacks; Sonnet 5.5 Hits ~4.87× vLLM Dollars-per-Dollar

Core Background & Industry Pain Points

Coding agents already patch repos and run experiments, yet reproducible tests of whether they can compress models for themselves or design their own serving stacks remain scarce. On 2026-11-04 CMU’s Infini-AI Lab posted CompressionBench and ServerBench on arXiv: the first asks agents to deliver executable compressed models under storage and time limits; the second asks them to design complete serving systems that pass latency and correctness checks.

Architecture Highlights & Internals

CompressionBench casts compression as a terminal task: given source weights, package/resident storage caps, a GPU and a wall-clock budget, agents choose quantization/pruning/distillation and implement inference code. Search uses full-model perplexity feedback; a separate container then verifies likelihood, generation accuracy (retain ≥80% of BF16), and storage. ServerBench starts from a minimal Transformers server and replays real coding sessions via AgentX. The objective is dollars-per-dollar: latency-qualified token value divided by GPU dollars. Submissions include the server and session concurrency.

Authoritative Benchmarks & Measured Scores

All figures below are from the papers’ tables (not AICoder re-runs). CompressionBench: six agents (Sonnet 5.5, Opus 5.5, GLM 5.3, Kimi K3.1, MiniMax M3.1, DeepSeek V4.1) × six model/limit settings; 29/36 submissions pass; selecting the lowest-PPL design per task yields ~3.4% average PPL above BF16 and ~77.33% mean downstream accuracy vs ~79.56% for remasured BF16. ServerBench: 11 models × three deployments; 22/33 runs verify and 13 beat the strongest baseline; Claude Sonnet 5.5 reaches 2.97 dollars-per-dollar on Gemma E4B (1×L40), about 4.87× the vLLM reference; Opus 5.5 reaches 4.40 on Qwen3.5 9B.

Developer Hands-on Guide

Papers and code: arxiv.org/abs/2611.06858, arxiv.org/abs/2611.06841, github.com/Infini-AI-Lab/Compression-Bench and Server-Bench; blogs at infini-ai-lab.github.io. Use them to compare agents’ experimental judgment on compression and serving, not just static QA accuracy. Caveat: scores depend on the papers’ hardware, budgets and workload replays and do not transfer blindly; failures often stem from generation regressions or unverified execution paths.