Together AI Launches Together Compiler: Agents Continually Optimize Production Inference, Up to 2.6x Higher TPS
On November 4 Together AI launched Together Compiler (TKCC), which pairs a swarm of AI agents with a persistent inference engine to keep rewriting inference kernels for real traffic, models and hardware, as part of what it calls Polymorphic Inference. The company says that within 24 hours it reached up to 2.6x higher TPS on models such as Kimi K3, GLM 5.3, MiniMax M3.1-Flash and DeepSeek v4.1-Flash, up to 50% faster on customer workloads, and it already serves trillions of tokens a day in production.
Key Takeaways
- Together Compiler recasts inference optimization from a classic compiler problem into an agentic search: agents propose kernel changes, a persistent engine checks speed and correctness, and only verified changes reach production
- Together says that within 24 hours it reached up to 2.6x higher TPS on Kimi K3, GLM 5.3, MiniMax M3.1-Flash and DeepSeek v4.1-Flash, and up to 50% faster on customer workloads
- On InferenceX low-latency batch size 1 it reports 1,111 TPS on MiniMax M3.1-Flash and 1,205 TPS on DeepSeek v4.1-Flash (company-reported); SemiAnalysis said it validated one of the results
- The persistent engine hot-reloads candidate forward passes, cutting time to serving to 9 seconds versus 467 seconds for restarting an engine, with 252x higher experiment throughput than baseline agentic optimization systems
- It already serves trillions of tokens a day in Together's production stack, where 40+ models are now optimized by agents