AI Coding Live News

On November 4 Together AI launched Together Compiler (TKCC), which pairs a swarm of AI agents with a persistent inference engine to keep rewriting inference kernels for real traffic, models and hardware, as part of what it calls Polymorphic Inference. The company says that within 24 hours it reached up to 2.6x higher TPS on models such as Kimi K3, GLM 5.3, MiniMax M3.1-Flash and DeepSeek v4.1-Flash, up to 50% faster on customer workloads, and it already serves trillions of tokens a day in production.

Long-horizon LLM agents accumulate expanding interaction histories that saturate GPU KV-cache memory and choke self-attention throughput. While sparse attention algorithms mitigate these bottlenecks mathematically, existing production serving engines (e.g. vLLM) are tightly coupled to dense tensor layouts, preventing heterogeneous sparse mechanisms from sharing infrastructure. Researchers from Harbin Institute of Technology (HIT) release SparseEngine (arXiv:2609.39068), an open-source, sparse-first inference engine built from the ground up for long-horizon agent workloads. SparseEngine establishes a shared lifecycle contract enabling 15 diverse sparse attention methods across four distinct families to control their custom KV representations while unifying memory management. It introduces Chain Cache to resume stateful KV-evicted contexts across multi-turn requests and Controllable Prefix-Cache Pruning to discard stale history without disrupting prefix matching. Empirical evaluations on long-horizon agent benchmarks show that SparseEngine delivers over 10x higher throughput under KV eviction, over 2.5x faster decoding throughput than vLLM at matched concurrency, and more than 2x end-to-end task speedups with zero degradation in model reasoning accuracy.

LM Studio announced a day-0 partnership with Inco to bring the Splash inference engine into LM Studio. They cite up to about 144 tok/s for Qwen3.8-27B on M5 Max, with setup docs at lmstudio.ai/blog/splash-engine. Splash was already open-sourced by Inco; this puts the Apple-silicon-optimized stack inside LM Studio so local desktop users do not have to assemble the engine separately.

Inco AI open-sourced Splash, a local inference engine built for Apple silicon and models like Qwen3.8-27B. On an M5 Max MacBook Pro it reports about 144 tok/s—up to 3× Ollama, 2× oMLX, and nearly 4× when an agent fans out sub-agents. LM Studio shipped day-0 support under Settings > Runtime > Experimental backends.

Meta overhauled the inference engine for Muse Spark 1.3, combining sub-quadratic linear attention with lossless parallel diffusion decoding. The upgrade delivers a 3.8x throughput increase and 50% memory reduction during long-horizon agent coding workflows.

Perplexity (@Perplexity_AI) open-sources Lily, the local inference engine built for hybrid compute in Perplexity Computer. Lily is specialized for Qwen3.6-35B-A3B on Apple silicon so on-device compute does not bottleneck Computer tasks.

Fireworks announced general availability of its Training API and Fireworks Lab, aiming to make model specialization accessible beyond labs that already own training clusters. The API lets teams connect a custom Python training loop to Fireworks-managed distributed training and rollout infrastructure, keeping control of loss, reward, data, and environment. Compute is split into Serverless mode for fast experiments and Dedicated mode when scale, control, and GPU economics matter. The stack is built for reinforcement learning as a single train-sample-serve-retrain loop, with three hard requirements: rollout throughput, weight sync so fresh weights land on the rollout deployment every step, and correctness so trainer and inference engine agree on the same model. Fireworks Lab additionally embeds forward-deployed researchers and engineers to co-design or build a model. Named customers include Cursor, Harvey, Figma, Vercel, Cognition, Factory, and Heidi.