[{"data":1,"prerenderedAt":51},["Reactive",2],{"news-story-news-1792366853834-lzoen":3},{"id":4,"title_zh":5,"title_en":6,"summary_zh":7,"summary_en":8,"takeaways_zh":9,"takeaways_en":14,"content_zh":19,"content_en":20,"external_links":21,"source":28,"category":35,"impact_score":36,"views_count":37,"likes_count":38,"related_entities":39,"tags":43,"image_url":50},"news-1792366853834-lzoen","苏黎世联邦理工与港大破解 MoE 路由探索困境 MirrorMoE：自适应专家剪枝化解共享专家抑制效应，全尺度 PPL 显著下降","ETH Zurich and HKU Resolve MoE Routing Exploration Dilemma with MirrorMoE: Adaptive Suppression Counteracts Shared Expert Homogenization","现代混合专家（MoE）普遍结合“细粒度专家切分”与“共享专家”设计，但二者在底层产生致命冲突：共享专家均质化了 Token 表征，抑制了路由探索，导致组合膨胀的专家空间无法被充分利用。苏黎世联邦理工学院（ETH Zurich）与香港大学团队发表 MirrorMoE（arXiv:2610.22353，GitHub: BingcongLi/MirrorMoE），通过 Token 自适应抑制冗余专家收缩探索空间，在从 100M 到 2.4B 全尺度上实现显著的 PPL 下降与下游基准全线涨点。","Modern Mixture-of-Experts architectures increasingly combine fine-grained expert segmentation with shared experts. However, ETH Zurich and HKU researchers identified an 'Exploration Dilemma' (arXiv:2610.22353, GitHub: BingcongLi/MirrorMoE): shared experts homogenize token representations, suppressing routing exploration just as fine-grained segmentation combinatorially enlarges the routing space. MirrorMoE resolves this via token-adaptive suppression of unnecessary experts, cutting pretraining perplexity across all model scales.",[10,11,12,13],"苏黎世联邦理工学院与香港大学团队发表 MirrorMoE（arXiv:2610.22353），揭示现代 MoE 中共享专家导致表征均质化、抑制细粒度路由探索的根本困境","提出 Token 自适应专家抑制算法，在保持零额外计算与训练时间开销的前提下动态收缩探索空间","在 100M 至 2.4B 参数全尺度上全面降低预训练困惑度（2.4B 模型 PPL 从 21.17 降至 20.50）","在六大下游基准平均得分提升 1.64 个百分点，其中 BoolQ 准确率大幅提升 4.03 个百分点（58.53% vs 54.50%）",[15,16,17,18],"ETH Zurich and HKU introduced MirrorMoE (arXiv:2610.22353) to solve the exploration dilemma between shared experts and fine-grained routing","Employs token-adaptive expert suppression to shrink the combinatorial routing space without adding runtime latency","Consistently reduces pretraining perplexity across scales from 100M to 2.4B (slashing 2.4B PPL from 21.17 to 20.50)","Lifts downstream benchmark averages by +1.64 points on the 2.4B model, highlighted by a +4.03 pt boost on BoolQ (58.53% vs 54.50%)","### 核心背景与行业痛点\n在现代混合专家架构（如 DeepSeek-V3/V4、Qwen-MoE）的演进中，两大设计已成为行业标配：一是**细粒度专家切分（Fine-Grained Segmentation）**，即切分出 64 或 128 个小型路由专家以增强组合表达能力；二是**共享专家（Shared Experts）**，即固定若干专家全量参与每个 Token 计算以捕获通用知识。然而，苏黎世联邦理工学院（ETH Zurich）与香港大学团队在 **`arXiv:2610.22353`** 中首次揭示了二者之间深层的**“探索困境（Exploration Dilemma）”**：共享专家的恒定介入极大地降低了 Token 表征的多样性与方差，反向导致门控网络（Router）趋向保守，在指数级膨胀的路由组合空间中丧失了充分探索各专家分工的能力，使得细粒度专家的理论表达增益大打折扣。\n\n### 架构亮点与底层机制\n1. **路由探索空间的 Token 自适应收缩（Token-Adaptive Expert Suppression）**：\n   - `MirrorMoE` 提出了一种镜像投影与自适应抑制机制。模型在门控计算阶段根据当前 Token 的特征分布，动态识别并抑制对当前上下文冗余度高的冷门候选专家，在每一步局部缩小路由候选空间；\n2. **零训练延迟与算力开销（Zero Overhead Architecture）**：\n   - 该抑制机制具备对称的闭式解析形式，无需引入复杂的辅助多轮前向计算或沉重的正则化损失，迭代耗时（Iteration Time）与原始 Vanilla MoE 完全一致甚至因冗余梯度过滤略有缩减。\n\n### 权威 Benchmark 与实测跑分对比\n- **跨参数规模预训练困惑度测试（Pretraining PPL, Table 1）**：\n  - **`MoE-100M-A38M` (8/64/2)**：预训练 PPL 从 54.78 降至 **51.77（-3.01 PPL）**，单步迭代时间为 7844 ms vs 7871 ms；\n  - **`MoE-498M-A121M` (8/64/2)**：预训练 PPL 从 28.31 降至 **27.36（-0.95 PPL）**；\n  - **`MoE-625M-A100M` (8/128/2)**：预训练 PPL 从 28.46 降至 **27.84（-0.62 PPL）**；\n  - **`MoE-2.4B-A270M` (8/128/2)**：预训练 PPL 从 21.17 降至 **20.50（-0.67 PPL）**，单步迭代耗时从 25,711 ms 降至 25,635 ms。\n- **下游基准全线评测（Downstream Evaluation, Table 2）**：\n  - 在 `ARC-E`、`OBQA`、`HellaSwag`、`PIQA`、`BoolQ`、`SIQA` 六大下游任务平均分上，`MirrorMoE` 取得系统性增长：在 2.4B 模型上平均分从 45.23% 提升至 **46.87%（+1.64 pts）**，其中 `BoolQ` 暴涨 **+4.03 个百分点**（58.53% vs 54.50%），`ARC-E` 达 46.63%（vs 45.37%）。\n\n### 开发者实战落地与开箱指南\n- **开源代码获取**：官方已在 GitHub 开源全部实现（`https://github.com/BingcongLi/MirrorMoE`）。\n- **MoE 预训练与微调集成建议**：正在自研或微调稀疏 MoE（如融合了共享专家与 64/128 路由专家体系）的团队，可直接在门控 Router 的 Softmax 前置计算中集成 MirrorMoE 的自适应抑制掩码算子，在完全不增加显存与训练时间的前提下，直接兑现细粒度专家应有的表达潜力与下游收敛上限。","### 核心背景与行业痛点\nModern Mixture-of-Experts (MoE) architectures increasingly pair two dominant design choices: fine-grained expert segmentation (splitting layers into 64 or 128 smaller routed experts) and shared experts (activating dedicated experts unconditionally for every token). In **`arXiv:2610.22353`**, researchers from ETH Zurich and HKU revealed an unexpected **'Exploration Dilemma'**: while shared experts provide general knowledge, they significantly reduce the variance of token representations. This homogenization reduces routing entropy and discourages routers from exploring the exponentially enlarged combinatorial space created by fine-grained segmentation, capping the realized expressiveness of the model.\n\n### 架构亮点与底层机制\n1. **Token-Adaptive Expert Suppression (MirrorMoE)**:\n   - `MirrorMoE` introduces an adaptive suppression mechanism that dynamically prunes unpromising expert candidates on a per-token basis. By shrinking the local search space adaptively, it forces the router to explore nuanced distinctions among the remaining specialized experts;\n2. **Zero Training Latency and Computational Overhead**:\n   - Designed with symmetric algebraic projections, MirrorMoE requires no iterative forward-backward passes or complex auxiliary regularization losses, matching or even slightly improving raw per-step iteration runtime compared to vanilla baselines.\n\n### 权威 Benchmark 与实测跑分对比\n- **Pretraining Perplexity Across Scales (Table 1)**:\n  - **`MoE-100M-A38M` (8/64/2)**: Reduces PPL from 54.78 to **51.77 (-3.01 PPL)** with 7,844 ms vs 7,871 ms step time;\n  - **`MoE-498M-A121M` (8/64/2)**: Cuts PPL from 28.31 to **27.36 (-0.95 PPL)**;\n  - **`MoE-625M-A100M` (8/128/2)**: Drops PPL from 28.46 to **27.84 (-0.62 PPL)**;\n  - **`MoE-2.4B-A270M` (8/128/2)**: Slashes PPL from 21.17 to **20.50 (-0.67 PPL)** while lowering step time from 25,711 ms to 25,635 ms.\n- **Downstream Zero-Shot Evaluations (Table 2)**:\n  - Across `ARC-E`, `OBQA`, `HellaSwag`, `PIQA`, `BoolQ`, and `SIQA`, MirrorMoE delivers consistent gains. On `MoE-2.4B-A270M`, the downstream average score rises from 45.23% to **46.87% (+1.64 pts)**, led by a **+4.03 percentage point surge on BoolQ** (58.53% vs 54.50%) and **+1.26 pts on ARC-E** (46.63% vs 45.37%).\n\n### 开发者实战落地与开箱指南\n- **Open-Source Implementation**: The code is available at `https://github.com/BingcongLi/MirrorMoE`.\n- **Integration Strategy**: For teams training or fine-tuning sparse MoE models with shared experts, substituting the naive top-k gating with MirrorMoE's token-adaptive suppression operator reconciles exploration and exploitation without increasing step latency or memory footprint.",[22,25],{"title":23,"url":23,"type":24},"https://github.com/BingcongLi/MirrorMoE","github",{"title":26,"url":26,"type":27},"https://arxiv.org/abs/2610.22353","arxiv",{"platform":29,"author_name":30,"author_handle":31,"author_avatar":32,"tweet_url":33,"published_at":34},"twitter","ETH Zurich & HKU","@huggingface","https://unavatar.io/x/huggingface","","2026-10-18T23:40:53.834Z","Research",90,35,0,{"models":40,"agents":41,"vendors":42},[],[],[],[44,45,46,47,48,49],"MoE","MirrorMoE","ETH Zurich","Architecture","Routing","Pretraining","https://arxiv.org/html/2610.22353v1/SE_energy_iter726.png",1792931174610]