CorrGRPO Solves Multi-Reward Imbalance in GRPO: Pearson Correlation Normalization Boosts Code & Tool Calling
Group Relative Policy Optimization (GRPO) forms the reinforcement learning foundation for reasoning models like DeepSeek-R1 by normalizing group rewards to derive advantage estimates. However, practical LLM alignment requires balancing multiple competing rewards, such as code execution correctness, format compliance, tool-calling precision, and safety. Standard GRPO computes advantage normalization across aggregate variance, causing large-scale, highly correlated reward components to overshadow critical fine-grained signals. HKUST researchers propose CorrGRPO (Correlation-Normalized GRPO), which normalizes reward covariances into Pearson correlation coefficients. Across code generation, tool invocation, and agent safety benchmarks spanning 0.5B to 8B models, CorrGRPO systematically outperforms standard GRPO.
Key Takeaways
- Addresses multi-reward scale imbalances in DeepSeek-R1 style GRPO using Pearson correlation coefficient normalization
- Lifts HumanEval/MBPP code generation Pass@1 by 3.2 to 4.8 percentage points over standard GRPO baselines
- Balances performance and guardrails, lifting tool-calling completion by 5.4% and safety defense from 82.1% to 93.7%