LoopCD Slashes Looped Transformer FLOPs by 48%: Contrastive Recurrent Decoding Lifts AIME 2024 to 73.3%

Background and the Problem

Looped Transformers achieve exceptional parameter efficiency by recurrently iterating a single shared block across multiple loops, emulating deep architectures within a fraction of memory footprints. However, standard decoding utilizes exclusively the final loop's hidden representation, discarding all intermediate states. Traditional contrastive decoding requires maintaining a distinct smaller amateur model in memory, which doubles VRAM usage and kernel dispatch overhead.

Architecture and How It Works

LoopCD exploits the inherent computational progression of looped recurrence to perform training-free contrastive decoding:

1. Natural Internal Weak-Strong Pairing: Early recurrent passes represent weaker internal states of the exact same model, supplying aligned amateur representations without extra models.
2. Dual Decoding Variants: Implements LoopCD-Logits (contrasting final logits against early loops with minimal projection cost) and LoopCD-Hidden (operating directly in latent vector space with zero extra output head passes).
3. Recurrent Loop Pruning: Guided search enables the model to reach superior consensus with substantially fewer recurrent cycles.

Benchmarks and Measured Results

Evaluated across four looped Transformer architectures on challenging math and code benchmarks:

1. AIME 2024 Surges to 73.33%: LoopCD-Logits elevates Ouro-2.6B-Thinking's Pass@1 from 61.88% to 73.33% (+11.45 percentage points).
2. HumanEval Coding Gains: LoopCD-Hidden boosts Huginn's code generation Pass@1 from 22.56% to 31.71% at zero extra projection overhead.
3. 22.5% to 48.2% FLOPs Reduction: Halving the loop count under LoopCD still matches or exceeds full-depth baselines, cutting forward FLOPs by up to 48.2%.

Getting Started for Developers

LoopCD requires zero parameter updates and drops seamlessly into Hugging Face and vLLM generate loops. Engineers deploying lightweight recurrent foundation models on edge devices can achieve immediate latency reductions alongside double-digit accuracy gains.