[{"data":1,"prerenderedAt":48},["Reactive",2],{"news-story-news-1792384474854-5h4rx":3},{"id":4,"title_zh":5,"title_en":6,"summary_zh":7,"summary_en":8,"takeaways_zh":9,"takeaways_en":14,"content_zh":19,"content_en":20,"external_links":21,"source":25,"category":32,"impact_score":33,"views_count":34,"likes_count":35,"related_entities":36,"tags":40,"image_url":47},"news-1792384474854-5h4rx","CMU 与复旦奠基循环架构理论 Hugging MoE：三明治拓扑破解参数重用瓶颈，8B 循环 MoE 媲美 32B 深层前馈模型","CMU and Fudan Pioneer Looped Model Foundations with Huginn MoE: Sandwich Topology Matches 32B Feedforward Baseline with 8B Resident Weights","循环语言模型（Looped Language Models）通过在逻辑深度上反复复用物理 Transformer 块大幅缩减权重规模，但过往架构将循环位置、输入注入与隐状态设计深度纠缠。卡内基梅隆大学（CMU）与复旦大学联合发表 Looped Models Done Right 第一部分（arXiv:2610.23036）。团队提出“前奏-循环块-尾声（Prelude-Recurrent-Coda）”三明治架构，在 5000 亿 Token 严苛预训练中，仅含 8.0B 常驻/7.94 亿激活参数的 Huginn MoE 在 MMLU、GSM8K 与 MATH500 上全面追平 32.0B 常驻/32 亿激活的前馈 MoE 基线。","Looped language models reuse physical Transformer blocks across logical depth to compress parameter footprints, but existing approaches entangle recurrence placement, input injection, and recurrent-state topologies. CMU and Fudan researchers published Part I of 'Towards Looped Models Done Right' (arXiv:2610.23036). Introducing a Prelude-Recurrent-Coda sandwich architecture, an 8.0B-resident / 794M-active Huginn MoE trained on 500B tokens matches a 32.0B-resident / 3.2B-active feedforward reference across MMLU, GSM8K, and MATH500.",[10,11,12,13],"CMU 与复旦大学联合发表循环模型奠基之作（arXiv:2610.23036），首次系统解耦循环位置、输入注入与隐状态设计","提出“前奏-循环块-尾声（P8-R12^8-C8）”三明治拓扑架构 Huginn，打破传统均匀全量循环的表征崩溃瓶颈","将原则推广至循环 MoE 架构，在 5000 亿 Token 预训练中，8.0B 常驻/7.94 亿激活参数的 Huginn MoE 性能全面打平 32.0B 常驻/32 亿激活的前馈基准","常驻显存开销暴降 75%，为在 16GB~24GB 消费级显卡部署 30B 级高性能推理提供了坚实工程路线",[15,16,17,18],"CMU and Fudan University established Part I of 'Towards Looped Models Done Right' (arXiv:2610.23036), decoupling recurrence placement from state design","Proposes the Prelude-Recurrent-Coda sandwich architecture (Huginn) to prevent representational collapse in looped transformers","Demonstrates that an 8.0B-resident / 794M-active Huginn MoE trained on 500B tokens fully matches a 32.0B-resident / 3.2B-active feedforward reference","Slashes resident memory footprint by 75%, charting a clear path to run 30B-tier reasoning models on 16GB–24GB consumer GPUs","### 核心背景与行业痛点\n随着模型参数规模膨胀至数百亿，边缘计算设备与端侧智能体受限于物理显存容量（VRAM），极难直接运行大型模型。循环语言模型（Looped Models，如近期的 NVIDIA Dory 1.0、Ouro）通过在深度方向上将同一组物理 Transformer 层循环迭代执行多次，理论上能在极小显存占用下实现相当于超深层模型的表达能力。然而，现有的循环模型设计极度经验主义——循环块放在哪一层、每一轮是否重新注入原始 Token 输入、隐状态如何初始化，三者在既有研究中被严重缠绕，缺乏坚实的理论与严谨的受控消融。CMU 语言技术研究所（LTI）与复旦大学团队在 **`arXiv:2610.23036`** 中发表了系统性奠基之作 **`Towards Looped Models Done Right (Part I)`**。\n\n### 架构亮点与底层机制\n1. **前奏-循环-尾声三明治拓扑（Prelude-Recurrent Block-Coda Sandwich Architecture, `Huginn`）**：\n   - 证明了将模型完全做成统一大循环（如 Ouro 架构）并非最优解。团队提出了 `P8-R12^8-C8` 范式：由 8 层非重用前奏（Prelude）负责基础语法特征提取，中间 12 层物理块循环迭代 8 次（等效 96 层逻辑深度），最后经 8 层非重用尾声（Coda）投影输出；\n2. **输入注入（Input Injection）与隐状态初始化公理化界定**：\n   - 严谨证明了在循环迭代中必须显式保持前奏特征向量的持续加权注入（$W_\\theta$ 门控）；相较于直接在循环体中强行复用未对齐的原始 Token，前奏注入能彻底消除浅层梯度弥散；\n3. **扩展至稀疏专家混合（Looped MoE）**：\n   - 将三明治循环原则无缝迁移至 MoE 架构，构建了 **`Huginn MoE`**，每个循环块挂载 25 个物理专家并每次动态路由 Top-2。\n\n### 权威 Benchmark 与实测跑分对比\n在严格匹配的 **5000 亿 Token（500B Tokens）** 大规模预训练与评测中（Table 2, 5, 6）：\n- **常驻显存节省 4 倍（Table 6）**：\n  - 前馈基准 `D112 MoE` 包含 **32.0B 常驻参数 / 3.2B 激活参数**；\n  - `Huginn MoE` 物理层数仅 28 层（8 Prelude + 12 Recurrent + 8 Coda），仅包含 **8.0B 常驻参数 / 793.9M 激活参数**（参数量缩减为 1/4）；\n- **权威评测全线比肩（Table 5）**：\n  - 在 `MMLU`（5-shot）、`BBH-CoT`、`GSM8K` 与 `MATH500` 等复杂逻辑推理基准上，8.0B 常驻参数的 `Huginn MoE` 全面逼近并部分超越 32.0B 参数的前馈基线，且系统性超越同参数量的无三明治循环架构 `Ouro MoE`；\n  - 在 730M 参数密集型模型实验中（Table 2），`P8-R12^8-C8` 拓扑在 ARC-C、HellaSwag 等通用任务上展现出压倒性的参数收敛效率。\n\n### 开发者实战落地与开箱指南\n- **理论与技术路线启示**：正在研发手机端、车载或本地离线 Coding Agent 的工程团队，不必盲目追求千亿参数量；采用三明治循环拓扑（Prelude + Looped Block + Coda）设计 7B~8B 物理尺寸的循环 MoE，能在 16GB~24GB 消费级显卡上实现媲美 30B 级模型的长链路推理能力，直接打破端侧部署的显存墙。","### 核心背景与行业痛点\nAs parameter counts swell, local edge devices and on-device coding agents are bottlenecked by physical VRAM limits. Looped language models (such as NVIDIA Dory 1.0 or Ouroboros architectures) repeatedly cycle tokens through a compact set of shared physical Transformer blocks, mimicking the capacity of ultra-deep networks within a fraction of the parameter memory footprint. However, prior architectures empirical designs hopelessly entangled recurrence placement, input injection mechanics, and latent state representations. Researchers from CMU LTI and Fudan University presented Part I of **`Towards Looped Models Done Right` (`arXiv:2610.23036`)**, establishing the first principled framework for looped transformers.\n\n### 架构亮点与底层机制\n1. **Prelude-Recurrent-Coda Sandwich Topology (`Huginn`)**:\n   - Proves that making the entire model an unbroken loop is mathematically sub-optimal. Instead, the team proposes a `P8-R12^8-C8` sandwich: 8 untied prelude layers extract basic syntactic representations, a compact 12-layer recurrent block cycles 8 times (yielding 96 logical layers), followed by 8 untied coda layers for task decoding;\n2. **Input Injection and Recurrent-State Mechanics**:\n   - Demonstrates that continually injecting prelude-derived representations across recurrent passes prevents representations from collapsing and eliminates vanishing gradients;\n3. **Scaling to Looped Mixture-of-Experts (Looped MoE)**:\n   - Extends the sandwich topology to sparse MoEs (25 physical experts, Top-2 routing per step), establishing **`Huginn MoE`**.\n\n### 权威 Benchmark 与实测跑分对比\nTrained on a rigorous **500-billion-token (500B)** corpus (Tables 2, 5, 6):\n- **4x Memory Footprint Compression (Table 6)**:\n  - The feedforward reference `D112 MoE` contains **32.0B resident parameters / 3.2B active parameters**;\n  - `Huginn MoE` has only 28 physical layers, requiring just **8.0B resident parameters / 793.9M active parameters** (a 75% reduction in disk and VRAM weights);\n- **Matching Frontier 32B Performance (Table 5)**:\n  - Across MMLU, BBH-CoT, GSM8K, and MATH500, the 8.0B-resident `Huginn MoE` matches the 32B feedforward MoE while decisively outperforming naive unstratified loops (`Ouro MoE`);\n  - In dense 730M-parameter experiments (Table 2), the sandwich architecture delivers optimal Pareto efficiency across ARC-C, HellaSwag, and CommonsenseQA.\n\n### 开发者实战落地与开箱指南\n- **Architectural Reference**: Detailed derivations and configuration tables are provided in `arXiv:2610.23036`.\n- **Edge Deployment Strategy**: Teams targeting local on-device agents on consumer GPUs (16GB–24GB VRAM) can leverage the sandwich looped MoE pattern to achieve 32B-class reasoning from an 8B physical memory footprint.",[22],{"title":23,"url":23,"type":24},"https://arxiv.org/abs/2610.23036","arxiv",{"platform":26,"author_name":27,"author_handle":28,"author_avatar":29,"tweet_url":30,"published_at":31},"twitter","CMU LTI & Fudan University","@huggingface","https://unavatar.io/x/huggingface","","2026-10-19T04:34:34.854Z","Research",93,45,0,{"models":37,"agents":38,"vendors":39},[],[],[],[41,42,43,44,45,46],"Looped Models","MoE","CMU","Fudan","Architecture","Efficiency","https://arxiv.org/html/2610.23036v1/figs/loop_family_timeline.png",1792916990650]