On-Policy vs Off-Policy Distillation Dynamics: Cambridge Empirical Study Reveals Forward KL Invariance and Reverse KL Sensitivity

Background and the Problem

On-policy learning is widely heralded as a foundational breakthrough in post-training, praised for curbing catastrophic forgetting and yielding superior generalization compared to offline supervised fine-tuning (SFT). However, prevailing empirical literature conflates multiple training hyperparameters simultaneously—varying rollout policies, token-level divergence objectives, and learning rate dynamics concurrently—leaving the true mechanistic drivers of distillation success obscured.

Architecture and How It Works

Researchers from the University of Cambridge led by Mihaela van der Schaar present a rigorous controlled study (arXiv:2609.35259) across Llama-3 and Qwen-2.5 architectures:

1. Orthogonal Variable Disentanglement: Systematically isolates the standalone effects of rollout policies (along a continuous student-to-teacher spectrum), token-level KL divergence directions, and optimization learning rates.
2. Invariance of Forward-KL: Proves that Forward KL divergence is remarkably robust to rollout distributions, achieving stable and high performance whether fed by offline teacher rollouts or on-policy student rollouts.
3. Extreme Sensitivity of Reverse-KL: Demonstrates that Reverse KL is fragile, strictly demanding student-generated rollouts to avoid optimization divergence.
4. Learning Rates Govern Forgetting: Confirms that catastrophic forgetting and parameter update sparsity are primarily dictated by learning rate magnitudes and scheduler annealing rather than the on-policy nature of rollouts.

Benchmarks and Measured Results

Evaluated on reasoning tasks spanning scientific deduction, clinical medicine, and complex arithmetic:

1. Flat Performance Curves Across Rollout Spectrum: Forward-KL maintains invariant accuracy across the entire spectrum of teacher-student rollout ratios, refuting the dogma that on-policy data is unconditionally superior.
2. Fragile Out-of-Distribution Advantages: While on-policy rollouts provide an edge on hard out-of-distribution Countdown arithmetic, this benefit does not reliably survive subsequent RLVR alignment.
3. Universal Robustness: Findings hold consistently across unclipped gradients, sampled Monte-Carlo KL estimators, and extended multi-step reasoning traces.

Getting Started for Developers

This study yields crucial practical cost-savings for distillation workflows. Teams operating with constrained inference compute budgets can confidently deploy offline teacher traces paired with Forward-KL loss functions, achieving parity with on-policy distillation without incurring the massive overhead of continuous online student rollouts.