Resolving Training-Inference Mismatch in LLM RLVR: Calibrated Importance Sampling Eliminates Gradient Variance

Background and the Problem

/ Background & Pain Points Scalable reinforcement learning with verifiable rewards (RLVR) relies on disaggregated architectures: decoupled high-throughput inference engines (vLLM/SGLang) sample rollout trajectories at scale, while distributed training engines (Megatron-LM/PyTorch) compute backward policy gradients. Because kernel optimizations, numerical precisions, and operator fusions differ subtly between engines, they assign slightly divergent probabilities to identical tokens. Under standard importance sampling, these discrepancies introduce exploding second moments in policy ratios, destabilizing policy gradient convergence.

Architecture and How It Works

/ Architectural Highlights Researchers introduced Calibrated Importance Sampling (CIS):

1. Logit-Displacement Invariance: Identifies an empirical property where per-logit additive perturbations $\varepsilon_t$ before softmax remain invariant to token confidence levels;
2. Confidence-Aware Ratio Truncation: Maps a constant logit-displacement threshold back to dynamic importance-weight caps that naturally tighten as token confidence increases while preserving exploratory weights on uncertain tokens;
3. Bounded Second Moments: Mathematically replaces the unbounded variance of exact importance sampling with a constant upper bound, ensuring numerical stability throughout extended reasoning rollouts.

Benchmarks and Measured Results

/ Benchmark & Evaluation Rigorous testing across three Mixture-of-Experts (MoE) backbones and five mathematical reasoning benchmarks confirmed clear superiority:

• Benchmark Leadership: CIS attained the highest five-benchmark average accuracy across all evaluated MoE checkpoints compared to truncated importance sampling and PPO baselines;
• Reduced Exploration Bias: Preserves unbiased gradient signals on low-confidence exploratory tokens, whereas heuristic clipping over-penalizes novel reasoning steps;
• Zero Overhead: Incurs negligible computational latency, requiring only vectorized tensor clamping operations.

Getting Started for Developers

/ Developer Practical Guide

• Formal Proofs: Documented in arXiv preprint 2609.32444;
• RLVR Training Pipeline Adoption: Recommended as a drop-in replacement for naive probability ratio clipping in open-source RL frameworks (e.g., verl, TRL, OpenRLHF);
• Engineering Impact: Eliminates numerical instability and NaN loss spikes in disaggregated vLLM-plus-Megatron training clusters.