Shanghai AI Laboratory has open-sourced AV-GRPO (arXiv: 2609.29816, GitHub: zhiyuxu03/AV-GRPO), a modality-anchored online diffusion reinforcement learning framework for joint audio-video generation. Addressing the persistent audio-visual desynchronization, semantic mismatch, and high compute overhead in coupled diffusion models, AV-GRPO decouples multimodal preference optimization into manageable unimodal subproblems via modality-anchored rollouts and trajectory-locked frozen-tower RL. It delivers state-of-the-art cross-modal synchronization on JavisBench and VABench, substantially outperforming LTX-2.3.
Key Takeaways
- ✓Extends Group Relative Policy Optimization (GRPO) to continuous joint audio-visual diffusion reinforcement learning.
- ✓Implements modality-anchored rollouts and trajectory-locked frozen towers to resolve multimodal credit assignment bottlenecks.
- ✓Releases 5DAV, the first systematically decoupled 5-dimensional difficulty-controllable training dataset.
- ✓Outperforms LTX-2.3 on JavisBench and VABench, boosting cross-modal synchronization by 28.4% and semantic alignment by 19.6%.
- ✓Training scripts, inference recipes, LoRA checkpoints, and datasets published under open-source licenses on GitHub.
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Core Background & Industry Pain Points
Joint audio-video diffusion models represent the next frontier in multimodal generation, yet existing pipelines frequently suffer from jarring audio-visual desynchronization—such as delayed engine roar during high-speed vehicle maneuvers or misaligned lip movements. Directly porting diffusion reinforcement learning (RL) to joint audio-video towers introduces severe credit assignment ambiguities: heterogeneous acoustic and visual reward signals entangle, making it impossible to determine whether policy gradients should update the audio head or the visual backbone.
Architecture Highlights & Internals
Shanghai AI Laboratory introduces AV-GRPO, an online diffusion RL framework grounded in modality-anchored rollouts. By anchoring one modality trajectory while exploring candidate variations in the peer modality, the framework decouples joint preference optimization into distinct unimodal subproblems. Trajectory-locked frozen-tower optimization slashes training compute, while the accompanying 5DAV dataset introduces five difficulty-controlled dimensions (motion velocity, reverberation, occlusions). Dynamically tuned noise perturbations accommodate the divergent temporal dynamics of audio waveforms and visual latents.
Authoritative Benchmarks & Measured Scores
Evaluated on JavisBench and VABench benchmarks: AV-GRPO outperforms LTX-2.3 across semantic alignment and physical plausibility, driving a 28.4% gain in cross-modal synchronization and a 41.2% reduction in temporal offset errors. Decoupled optimization cuts peak VRAM consumption during RL backpropagation by 52%, enabling single-GPU LoRA fine-tuning.
Developer Hands-on Guide
Researchers can clone https://github.com/zhiyuxu03/AV-GRPO to access training recipes, download pre-trained LoRA weights from Hugging Face, and inspect the 5DAV evaluation dataset.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.