Negative Advantage Is a Double-Edged Sword: Calibrating Advantage in GRPO for Deep Search

`).

2. Methodology: CalibAdv

The authors propose CalibAdv, a three-pronged advantage calibration framework designed to mitigate mis-penalization and rebalance training signals.

A. Soft Advantage Penalization for Intermediate Steps

To address the misattribution of penalties, CalibAdv introduces a fine-grained mechanism to adjust negative advantages based on the correctness of intermediate retrieval steps.

• Silver Documents: For a given question, documents retrieved by correct rollouts are treated as "silver documents" (ground truth proxies).
• Correctness Score ($c_s$): For an intermediate step in an incorrect rollout, the correctness score is calculated as the proportion of retrieved documents that overlap with the silver documents.
• Attenuation: If an intermediate step has a negative advantage ($A < 0$) but a high correctness score, the magnitude of the penalty is scaled down:
  $$\tilde{A}_s = A_s \cdot (1 - c_s)$$
  This ensures that useful search actions are not heavily penalized just because the final answer was incorrect.

B. Advantage Rebalancing for Final Answer Step

To combat training collapse caused by the dominance of negative signals, the method rebalances the ratio of positive to negative advantages specifically at the final answer step.

• Ratio Calculation: For each group of rollouts, the ratio ($r_g$) of the absolute value of negative advantages to positive advantages is computed.
• Scaling: The positive advantages are scaled up by this ratio (controlled by a hyperparameter $\lambda$) to restore balance:
  $$\tilde{A}^+_g = \lambda \cdot r_g \cdot A^+_g$$
  This prevents the model from being overwhelmed by negative signals, which drives the entropy collapse.

C. Decoupling Special Tokens

The authors observe that the <think> token (used to delimit reasoning) is a frequent victim of advantage imbalance, leading to format collapse.

• Strategy: Instead of requiring the model to generate <think>, it is prepended to the prompt as a fixed prefix.
• Effect: Since the model does not generate this token, no advantage signal is assigned to it, preventing probability fluctuations that could trigger format degradation.

3. Key Contributions

1. Diagnosis of GRPO Failures: The paper provides empirical evidence linking training collapse in deep search to the dominance of negative advantages and the coarse-grained penalization of correct intermediate steps.
2. CalibAdv Framework: A novel, annotation-free method that calibrates advantages using internal signals (silver documents derived from correct rollouts) rather than requiring external LLM labeling or sub-question annotations.
3. Stability and Performance: The method successfully prevents catastrophic training collapse while significantly improving question-answering performance across diverse model sizes and architectures.

4. Experimental Results

The authors evaluated CalibAdv on three models (Qwen2.5-3B, Qwen2.5-7B, Llama-3.2-3B) across seven benchmarks (including HotpotQA, 2WikiMultiHopQA, Natural Questions, etc.).

• Performance Improvement: CalibAdv achieved an average 11.80% relative improvement in F1 score compared to standard GRPO (Search-R1).
• Training Stability:
  • Standard GRPO and baselines (like SimpleTIR, LLD) often suffered from training collapse (indicated by a sharp drop in performance or high Perplexity) within 50–200 steps.
  • CalibAdv successfully trained to completion ("No collapse") on all models and datasets.
  • It reduced the "High PPL Ratio" (outputs with Perplexity > 50, indicating nonsense) to 0.00%.
• Ablation Studies:
  • Removing the Soft Penalization led to a drop in F1 score and a higher rate of mis-penalized steps.
  • Removing Advantage Rebalancing resulted in the return of training collapse and high PPL ratios.
  • Removing the Decoupled Token caused format instability.
• Efficiency: The "Silver Document" proxy used for correctness scoring was found to be highly reliable (89% human verification) and significantly more cost-effective than using an external LLM (DeepSeek-V3.2) for step evaluation, saving ~67% wall-clock time and 200% GPU cost.

5. Significance

This work highlights that while negative advantages are crucial for RLVR, they act as a "double-edged sword" in multi-turn tasks. If not carefully calibrated, they can destroy a model's natural language capabilities.

• Practical Impact: CalibAdv enables the robust training of deep search agents without the need for expensive intermediate annotations or complex tree-structured sampling.
• Generalizability: The approach is model-agnostic and effective across different parameter sizes, suggesting that advantage calibration is a fundamental requirement for scaling RL in complex, multi-step reasoning tasks.
• Future Direction: It shifts the paradigm from simply "penalizing wrong answers" to "calibrating the learning signal" to ensure that correct intermediate behaviors are reinforced even in the presence of final errors.