Domain-Normalized MOPD: Rescaling Teacher Feedback Prevents Instruction Dominance in Multi-Expert Distillation

Background and the Problem

/ Background & Pain Points Reinforcement learning easily produces domain-specific specialist foundation models (e.g., mathematics, software engineering, instruction alignment), yet real-world applications demand unified generalist systems. Multi-Teacher On-Policy Distillation (MOPD) routes prompts to corresponding specialists whose token-level feedback guides a unified student model. However, standard MOPD students frequently underperform individual specialists and suffer catastrophic capability degradation on mathematical and algorithmic reasoning.

Architecture and How It Works

/ Architectural Highlights Researchers from SUTD and NTU uncovered the underlying mechanism and introduced Domain-Normalized MOPD (DN-MOPD):

1. The Instruction Dominance Trap: Discovers that supervisory feedback from instruction-following teachers exhibits several times higher variance and dispersion than feedback from mathematical specialists, overwhelming the student's shared parameter updates and washing out specialized reasoning priors;
2. Variance-Normalized Gradient Rescaling: Retains domain routing while dynamically normalizing teacher feedback by its empirical distribution spread, scaling down overly dispersed instruction feedback to establish parity with dense mathematical gradients;
3. Targeted Calibration: Proves that successful multi-specialist fusion requires calibrating feedback intensity rather than merely routing prompt assignments.

Benchmarks and Measured Results

/ Benchmark & Evaluation Extensive experiments on Qwen3.5 checkpoints across three model tiers demonstrated decisive advantages:

• Consistent Cross-Benchmark Superiority: Outperforms conventional MOPD on six public benchmarks across all parameter sizes, three distinct random seeds, and both tight and generous output token constraints;
• Full Mathematical Capability Recovery: Successfully preserves deep mathematical logic alongside conversational alignment, recovering gains previously lost during unnormalized multi-teacher runs;
• Convergence Stability: Suppresses training instability and loss spikes caused by conflicting cross-domain gradient magnitudes.

Getting Started for Developers

/ Developer Practical Guide

• Preprint Citation: Detailed in arXiv preprint 2609.35347;
• Training Recipe Adoption: Teams employing multi-teacher distillation in frameworks like TRL and OpenRLHF should introduce domain-variance scaling factors before backward passes;
• Zero Compute Overhead: Implementation requires only scalar normalization during loss aggregation with zero impact on training throughput.