SLCA-GRPO: Segment-Locked Credit Assignment in Tool-Calling RL Resolves Reward Contamination

Background and the Problem

/ Background & Pain Points Tool-calling agents produce heterogeneous outputs, interleaving structured tool invocations with conversational natural language summaries. In standard on-policy Reinforcement Learning algorithms such as GRPO, trajectory-level scalar advantages are broadcast indiscriminately across all tokens. Consequently, gradient noise from free-form summary generation leaks into critical tool-selection tokens, creating Cross-Segment Credit Misattribution (CSCM) and optimization instability.

Architecture and How It Works

/ Architectural Highlights To resolve this structural flaw, researchers introduce SLCA-GRPO, defined by three primary components:

1. Segment-Locked Credit Assignment (SLCA): Decouples advantage estimation at structural segment boundaries within a single group of rollouts, eliminating the need for expensive intermediate branching rollouts;
2. Schema-Guided LLM Simulator (SGLS): Serves as deterministic foundational training infrastructure, bypassing expensive and non-deterministic real API invocations;
3. Hierarchical Rewards (HierR): Routes objective execution advantages exclusively to tool tokens and conversational preference advantages to summary tokens, completely cutting off the dominant cross-segment contamination channel.

Benchmarks and Measured Results

/ Benchmark & Evaluation Evaluated on a 7B backbone against standard GRPO, ToolPO, and RLTR under identical training budgets:

• In-Domain Accuracy: Outperforms baselines by +2.53 pp with faster convergence;
• Berkeley Function-Calling Leaderboard (BFCL): Improves generalization performance by +1.36 pp;
• τ²-Bench: Achieves a dramatic +9.15 pp gain in multi-step nested agent environments;
• Efficiency: Reduces redundant tool invocations and overall API execution overhead by 18.4%.

Getting Started for Developers

/ Developer Practical Guide The complete codebase and datasets are fully open-source:

• GitHub Repository: Accessible at https://github.com/SLCA-GRPO/SLCA-GRPO with PyTorch & TRL integration;
• Dataset: Available at YanZhanPKU/SLCA-GRPO-Datasets on Hugging Face;
• Best Practices: Enable segment decoupling and assign hierarchical loss scaling (0.7 tool / 0.3 summary) to guarantee stable convergence during agent post-training.