Hugging Face Ships Ultimate MoE Training Handbook: ~500k–800k GPU-Hours Distilled into a Megatron-LM Guide
On 2026-11-05 Hugging Face released the free Ultimate MoE Training Handbook. Co-founder Thomas Wolf cited ~800k GPU-hours, ~70k words and 55 interactive figures; author Nouamane Tazi separately cited ~500k GPU-hours. The Megatron-LM guide covers routing, expert sizing, communication and debugging via the official Space.
Key Takeaways
- Official: Hugging Face Space “The Ultimate MoE Training Handbook” (2026-11-05)
- Thomas Wolf (HF co-founder) on X: ~800k GPU-hours, ~70k words, 55 interactive figures
- Author Nouamane Tazi separately cited ~500k GPU-hours; prefer the Space body over a reconciled figure
- Covers Megatron-LM: routing, expert sizing, all-to-all, aux vs loss-free bias, upcycling and serving
- Free and open; small teams can follow the runbook flag-by-flag against logs









