[{"data":1,"prerenderedAt":48},["Reactive",2],{"news-story-news-1792368356004-gruu6":3},{"id":4,"title_zh":5,"title_en":6,"summary_zh":7,"summary_en":8,"takeaways_zh":9,"takeaways_en":14,"content_zh":19,"content_en":20,"external_links":21,"source":25,"category":32,"impact_score":33,"views_count":34,"likes_count":35,"related_entities":36,"tags":40,"image_url":47},"news-1792368356004-gruu6","小红书、北大与港中文深圳提出 LangDiT：专为连续扩散语言模型重构 Transformer，生成困惑度骤降至 11.33 刷新 SOTA","Xiaohongshu, PKU, and CUHK Shenzhen Introduce LangDiT: Re-Engineering Transformers for Continuous Diffusion Language Models Slashes GenPPL to 11.33","现有连续扩散语言模型（CDLM）普遍直接套用自回归 LLM 或图像 DiT 架构，存在特征尺度失控与梯度路由紊乱问题。小红书、北京大学、香港中文大学（深圳）与 UCLA 团队联合发表 LangDiT（arXiv:2610.22330）。通过骨干网与 Q/K RMSNorm、展宽噪声条件调制、词表头残差截断解耦及置信度门控自条件引导，在 OpenWebText 基准上将 LangFlow 生成困惑度压至 21.72，适配 Plaid 架构后进一步取得 11.33 的全新 SOTA 困惑度记录。","Continuous diffusion language models (CDLMs) currently transplant architectures from causal LLMs or vision DiTs that are not optimized for parallel diffusion. Researchers from Xiaohongshu (Dots Studio), Peking University, CUHK Shenzhen, and UCLA introduced LangDiT (arXiv:2610.22330). By separating feature normalization from noise modulation, detaching shortcut gradients, and introducing confidence-gated self-conditioning, LangDiT reduces generation perplexity (GenPPL) to 21.72 on LangFlow and 11.33 on Plaid across 1024-token OpenWebText rollouts.",[10,11,12,13],"小红书、北大与港中文深圳联合发表 LangDiT（arXiv:2610.22330），首次系统重构专属于连续扩散语言模型的 Transformer 架构","通过 Q/K RMSNorm、展宽噪声条件调制解耦数值稳定性与时间步注入，利用词表头梯度截断消除自条件反馈污染","在 OpenWebText 1024 长度无条件文本生成中，将 LangFlow 生成困惑度降至 21.72，将 Plaid 困惑度大幅压缩至 11.33 的全新纪录","在仅需 128 步去噪采样下仍取得 27.51 的优异困惑度，证实骨干网络定制是非自回归语言模型取得质变的核心动力",[15,16,17,18],"Xiaohongshu, PKU, and CUHK Shenzhen introduced LangDiT (arXiv:2610.22330), the first transformer backbone customized for continuous diffusion language models","Decouples representation scaling from noise modulation via Q/K RMSNorm and stabilizes training through vocabulary shortcut detachment","Drives 1024-token generation perplexity on OpenWebText down to 21.72 on LangFlow and a record 11.33 on Plaid formulations","Maintains strong quality (27.51 GenPPL) even under 128 denoising steps, isolating transformer design as the core lever for diffusion text generation","### 核心背景与行业痛点\n连续扩散语言模型（Continuous Diffusion Language Models, CDLM，如 `LangFlow`、`Plaid`）作为并行文本生成的代表性前沿范式，理论上能够打破自回归解码的逐 Token 串行瓶颈，实现极高吞吐的批量内容合成。然而，学术界与工业界现有的 CDLM 架构绝大多数只是简单搬用传统的自回归 Transformer（如 LLaMA）或图像扩散模型（如 DiT），未曾针对离散文本在连续隐空间中的扩散动力学做底层架构适配，导致生成文本困惑度（GenPPL）居高不下、长序列扩散采样极易崩溃发散。为此，小红书（Dots Studio）、北京大学、香港中文大学（深圳）与 UCLA 联合推出了首个专为语言扩散定制的骨干网络 **`LangDiT`（Language Diffusion Transformer，`arXiv:2610.22330`）**。\n\n### 架构亮点与底层机制\n1. **解耦特征尺度控制与噪声依赖调制（Decoupled Feature & Noise Control）**：\n   - 在骨干网络层与 Query/Key 注意力前置引入独立 **RMSNorm**，将文本特征本身的数值稳定性与扩散时间步的噪声注入解耦；同时展宽噪声条件向量维度（Condition Width 从 128 提升至 256），避免高噪声强度下语义表征被淹没；\n2. **词表头残差路径截断与可学习边界校准（Shortcut Detachment & Bounded Calibration）**：\n   - 针对连续隐向量向离散词表映射时的快捷通道（Shortcut），在反向传播时对其正向贡献与 Codebook 梯度进行显式解耦阻断，避免自条件反馈污染初始嵌入先验；\n3. **置信度门控训练自条件引导（Confidence-Gated Self-Conditioning）**：\n   - 训练期间引入基于信息熵的置信度门控，仅对高置信度隐状态进行自条件注入（Bootstrap），而在推理期直接使用无加权后验均值估计，兼顾采样稳定性与文本多样性。\n\n### 权威 Benchmark 与实测跑分对比\n在 **OpenWebText 1024 Token 无条件长文本生成基准** 上对两种主流 CDLM 公式进行了全面评估：\n- **`LangFlow` 架构适配**：\n  - 在 1024 步扩散（NFE）下，生成困惑度（GenPPL ↓）达到 **21.72**（刷新同类模型纪录）；\n  - 在仅 128 步快速采样（NFE）下，保持 **44.72 GenPPL** 的稳定可用水平；\n- **`Plaid` 架构适配（Table 2）**：\n  - 将 `LangDiT` 的组件移植至 Plaid 架构后，在 1024 步采样下创下 **11.33 GenPPL** 的历史最佳纪录；\n  - 在 128 步低延迟采样下，GenPPL 达到 **27.51**，在保持原有隐空间表征与训练目标完全不变的情况下，纯靠架构改良实现了生成质量质的飞跃。\n\n### 开发者实战落地与开箱指南\n- **论文与技术细节**：完整推导与消融实验详见 `arXiv:2610.22330`。\n- **工程选型启示**：探索非自回归并行大模型、实时语音/文本联合扩散生成的工程团队，应避免直接使用标准 LLaMA 或 DiT 仓库改写；引入 LangDiT 的 Q/K RMSNorm 与词表头梯度解耦策略，是稳定连续语言扩散训练并压低生成困惑度的关键基础设施。","### 核心背景与行业痛点\nContinuous diffusion language models (CDLMs, such as `LangFlow` and `Plaid`) represent a promising alternative for non-autoregressive parallel text generation, offering high-throughput generation free of token-by-token latency constraints. However, virtually all existing CDLMs directly transplant architectures from causal LLMs or vision Diffusion Transformers (DiTs). These generic models suffer from feature scale collapse and gradient routing conflicts when diffusing discrete text tokens embedded in continuous manifolds. Researchers from Xiaohongshu (Dots Studio), Peking University, CUHK Shenzhen, and UCLA designed **`LangDiT` (`arXiv:2610.22330`)**, the first dedicated transformer architecture optimized specifically for continuous diffusion language modeling.\n\n### 架构亮点与底层机制\n1. **Decoupling Feature Normalization from Noise Modulation**:\n   - Introduces backbone RMSNorm and headwise Q/K RMSNorm alongside a wider noise conditioning projection (doubled from 128 to 256), decoupling intrinsic representation scaling from stochastic noise updates;\n2. **Vocabulary Shortcut Detachment & Bounded Calibration**:\n   - Separates the forward shortcut contribution from direct codebook gradient routing at the output vocabulary head, preventing self-conditioning feedback from corrupting base embeddings;\n3. **Confidence-Gated Training Self-Conditioning**:\n   - Gated bootstrap inputs filter training steps by entropy confidence, ensuring that only reliable latent estimates are injected during self-conditioning while unweighted posterior-mean estimates are used at inference.\n\n### 权威 Benchmark 与实测跑分对比\nEvaluated on **OpenWebText unconditional generation of 1024-token sequences**:\n- **`LangFlow` Formulation**:\n  - Achieves a state-of-the-art generation perplexity (**21.72 GenPPL**) at 1024 denoising NFEs, and **44.72 GenPPL** with just 128 NFEs;\n- **`Plaid` Formulation (Table 2)**:\n  - Slashes generation perplexity to **11.33 GenPPL** at 1024 NFEs, and **27.51 GenPPL** at 128 NFEs;\n  - These gains isolate the Transformer architecture itself as the fundamental driver of performance, achieved without altering the underlying training objectives or latent representations.\n\n### 开发者实战落地与开箱指南\n- **Research Reference**: Full architectural specifications are detailed in `arXiv:2610.22330`.\n- **Engineering Takeaways**: Teams exploring non-autoregressive generation, parallel speech synthesis, or diffusion-based code infilling should integrate LangDiT's Q/K normalization and vocabulary shortcut detachment to eliminate gradient instability and mode collapse.",[22],{"title":23,"url":23,"type":24},"https://arxiv.org/abs/2610.22330","arxiv",{"platform":26,"author_name":27,"author_handle":28,"author_avatar":29,"tweet_url":30,"published_at":31},"twitter","Dots Studio (Xiaohongshu) & PKU / CUHK Shenzhen","@huggingface","https://unavatar.io/x/huggingface","","2026-10-19T00:05:56.004Z","Research",91,34,0,{"models":37,"agents":38,"vendors":39},[],[],[],[41,42,43,44,45,46],"LangDiT","Diffusion","Non-Autoregressive","Architecture","Xiaohongshu","NLP","https://arxiv.org/html/2610.22330v1/final_architecture.svg",1792931174029]