Ai2 ships Olmo-core 3: open trillion-scale MoE training stack; 858 TFLOP/s/GPU at 1.2T on B300
On 2026-10-01 Ai2 released Olmo-core 3, an open MoE training stack (GitHub, PyPI ai2-olmo-core). It moves from FSDP to DDP + expert parallelism; on 8×B300 a 47B MoE hits ~2.7× throughput (52k vs 19.4k tok/s/GPU). Systems benches include 1.2T on 512 GPUs at up to 858 useful-model TFLOP/s/GPU and a DeepEP v2 capacity probe at 2.38T. Report: olmocore3.

Key Takeaways
- Announced 2026-10-01: allenai.org/blog/olmocore3 + HF mirror + github.com/allenai/OLMo-core
- Stack: EP/PP/distributed optimizer; NVSHMEM rowwise EP + GPU-resident routing + grouped GEMM; optional DeepEP v2
- Throughput: expert pool 8→128 at ~3.2B active, <5% drop; 47B MoE on 8×B300 52k vs 19.4k tok/s/GPU (~2.7×)
- Scale: 1.2T / 58.36B active / 512 GPUs up to 858 TFLOP/s/GPU (random-routing systems ref); DeepEP v2 probe 2.38T
- MXFP8 ~+21% vs BF16, peak active mem 103→95 GiB; topology-agnostic checkpoints; next Olmo will be MoE
