The Teacher Is a Direction, Not a Destination: RIDE Extrapolates RL Representation Residuals to Surpass Teacher Models

Background and the Problem

On-policy distillation (OPD) trains student policies by aligning next-token distributions on student-sampled rollouts with an RL-privileged teacher. While generalized variants seek to allow students to surpass teachers via output-space reward extrapolation, they are hindered by intrinsic mechanics: language-model projection heads attenuate representation shifts anisotropically, and token-level log-probability ratios severely amplify sampling noise, inducing training divergence.

Architecture and How It Works

Researchers introduce RIDE (RL-Induced Direction Extrapolation, arXiv:2609.36484), establishing that 'the teacher is a direction, not a destination':

1. Geometric Representation Shift: Demonstrates that reinforcement learning applies consistent geometric representation shifts across layers relative to base model weights.
2. Layer-Wise Residual Extrapolation: RIDE calculates the vector residual between the RL teacher and base model across all hidden states and token steps, regressing student representations toward targets extrapolated beyond the teacher along this directional vector.
3. Directional Linear Reward with Quadratic Penalty: Formulates the regression objective as maximizing a directional reward under a quadratic constraint centered on the teacher, enforcing stability while steering the student onward.
4. Circumvention of LM-Head Attenuation: Operates entirely within representation space, eliminating logit noise amplification.

Benchmarks and Measured Results

Evaluated across four diverse base/RL-teacher pairs spanning distinct architectures and pretraining genealogies:

1. Consistently Surpasses RL-Trained Teachers: RIDE matches or outperforms the RL-trained teacher across every evaluated model pair, standing as the only method whose mean performance surpasses the privileged supervisor.
2. Overcomes Output-Space Collapse: Where output-space extrapolation degrades student performance when the teacher remains close to base weights, RIDE maintains consistent positive transfer.
3. Robust Reasoning Generalization: Delivers higher precision on multi-step reasoning rollouts without hallucination degradation.

Getting Started for Developers

RIDE is open-sourced at https://github.com/xixixixixxxx/RIDE. For ML engineers optimizing post-training pipelines and compact reasoning models, replacing output-space KL divergence with RIDE representation residual extrapolation offers an elegant pathway to create student models that outshine their RL teachers without executing expensive secondary RL loops.