AI Coding Live News

Researchers from Tsinghua University and CUHK introduced a breakthrough paper titled 'Cache Engineering for Agents' (arXiv:2610.28577) alongside the open-source AKV serving engine (thu-nics/AKV). While multi-turn agents rely on traditional 'context engineering' (text trimming and summarization) that invalidates prefix KV-caches and incurs punishing re-prefilling overhead, cache engineering empowers agents to directly manipulate KV-cache states via 'drop' and 'repos' primitives. Coupled with AKV's drop-aware radix-tree engine, the framework slashes agent inference costs by 27.7%, accelerates P95 latency by 1.5x, and boosts benchmark solve scores by 1.7 points.

Zoom

LLM-guided evolutionary algorithms (such as AlphaEvolve) have achieved breakthrough results in computational optimization, including circle packing and systems tuning. However, existing methods optimize strictly over iteration counts rather than fiscal expenditure, burning dozens of dollars per task. Researchers from NUS, Stanford, and HKUST present FrugalEvo (arXiv:2610.03675), a cost-aware program evolution framework maximizing gain per dollar. FrugalEvo implements an asymmetric dual-LLM division of labor: a powerful, higher-cost LLM explores high-level solution concepts, while a cheap, high-speed LLM generates and iteratively refines concrete implementations. Additionally, FrugalEvo engineers prompt harnesses to maximize prefix KV-cache reuse across evolutionary generations. Across 10 mathematical optimization problems and ALE-Bench-Lite suites, FrugalEvo matches or outperforms OpenEvolve, EvoX, and SwarmResearch. On circle packing, FrugalEvo achieves SOTA performance using GLM-5.3/Flash for only $0.55 and GPT-5.6 Terra/Luna for $1.68—slashing costs by up to 99% compared to conventional $50 multi-agent baselines.

Zoom

Chunked KV-cache compression reduces the memory and computational footprints of long-context LLMs by condensing contiguous token windows into fewer cache entries at fixed strides. Researchers from Stanford and Renmin University uncover an intrinsic vulnerability: chunking introduces a new relative positional coordinate—a token's phase relative to compression-window boundaries. The study reveals systematic 'phase sensitivity,' where retrieval accuracy for identical facts varies by up to 40 percentage points depending solely on input token phase alignments. Pretraining transformers across various KV compression configurations and conducting causal interventions, the authors demonstrate that standard aggregate benchmark scores mask recurrent blind spots, establishing a new evaluation standard for long-context compression.

Zoom

Deploying 70B+ LLMs under tight memory constraints necessitates extreme 4-bit weight, activation, and KV-cache quantization (W4A4KV4). However, current methods predominantly focus on suppressing activation outliers, overlooking their geometric alignment with the quantizer grid. Researchers introduce PrismQuant, a quantizer-aware rotation framework that mathematically aligns the leading activation eigenspace with the constant group subspace of asymmetric grouped INT4 quantization. Formulated as a Ky Fan trace maximization problem, PrismQuant derives a closed-form, provably optimal rotation solution without requiring expensive gradient descent. Benchmarked across Llama, Qwen, and Mistral architectures up to 70B parameters and 30B MoE variants, PrismQuant sets new SOTA marks: on Llama-3.1-70B under W4A4KV4, it achieves 72.46% zero-shot accuracy—merely 0.22 percentage points below full FP16 precision. Deployment benchmarks demonstrate a 1.51x prefill speedup and 56.34% reduction in peak decoding VRAM with code open-sourced on GitHub.

Zoom

While looped language models enable depth-adaptive inference by executing fewer layer repetitions on easy tokens and more on complex tokens, variable loop counts break conventional batching engines like vLLM. Researchers from TUM and Imperial College introduced Continuous Depth Batching (CDB). By dynamically reorganizing batches between loop iterations, orchestrating looped KV-caches, and asynchronously predicting token exits, CDB achieves up to 99% of the theoretical maximum inference speedup across Ouro 1.4B and Huginn 3.5B architectures.

Zoom

NVIDIA (@nvidia) has open-sourced SoL-Pi, a high-throughput inference acceleration engine tailored for autonomous coding agents. Addressing the severe computational redundancy in multi-turn AST parsing and context diffing, SoL-Pi leverages speculative tree decoding and dynamic KV-cache branch reuse to cut token traffic by 44%-49% while lowering inference API costs by 33%.

Zoom