Technical Summary: Closing the Reflection Gap with RefGRPO

1. Problem Statement: The Reflection Gap

Large Language Models (LLMs) are increasingly deployed as agents that interact with external environments, receiving concrete feedback such as execution results, error messages, and tool outputs. A competent agent should leverage this feedback to accurately assess its own performance. However, the authors identify a persistent "reflection gap": LLM agents frequently mis-assess their own outputs even after observing concrete environment feedback.

Contrary to the well-documented issue of overconfidence, the authors find that agents often exhibit significant underconfidence, flagging correct answers as wrong. Standard Reinforcement Learning (RL) algorithms, specifically Group Relative Policy Optimization (GRPO), fail to resolve this. While standard RL improves task accuracy, it leaves the agent's post-feedback reflection quality broken.

The Root Cause: The authors attribute this to a credit-assignment mismatch in outcome-only RL. In standard GRPO, the advantage signal is derived solely from the final task outcome ($r$). Consequently:

• Honest Error Flags: If an agent correctly flags an error ($s_{ref}=0$) but the outcome is failure ($r=0$), the negative advantage trains the agent to suppress correct error flags.
• Underconfident Flags: If an agent incorrectly flags a success as an error ($s_{ref}=0$) but the outcome is success ($r=1$), the positive advantage trains the agent to doubt its own successes.
  Standard RL optimizes for task correctness but inadvertently penalizes honest self-assessment.

2. Methodology: RefGRPO

To close this gap, the authors propose RefGRPO, an algorithm that augments standard RL with two key components: a free calibration bonus and a dynamic schedule.

A. Free Calibration Bonus

The core innovation is a bonus computed by contrasting the agent's post-feedback reflection ($s_{ref}$) with the actual outcome ($r$).

• Mechanism: For each rollout $k$, a binary calibration bonus $c_k = \mathbb{I}(s_{ref,k,H} = r_k)$ is calculated. This bonus is added to the outcome reward before group normalization:
  $$\tilde{r}_k = r_k + \alpha(t) \cdot c_k$$
• Significance: This bonus is "free" because it requires no additional reward model, LLM judge, or external annotation; it relies solely on the agent's own reflection and the known environment outcome. It ensures that well-calibrated rollouts (where the agent correctly predicts the outcome) receive a positive relative advantage regardless of whether the task itself succeeded or failed.

B. Dynamic Schedule

The authors observe that a static calibration bonus can slightly degrade task accuracy, as the reward signal is partially diverted to calibration quality. To balance this, they introduce a dynamic schedule for the coefficient $\alpha(t)$:

• Two-Stage Schedule:
  1. Front-loading ($t \le \gamma \cdot T$): A larger coefficient ($\alpha_0$) is used to prioritize learning calibration early in training.
  2. Decay ($t > \gamma \cdot T$): The coefficient decays to a smaller value ($\alpha_1$, often 0), allowing the model to focus on maximizing task performance while retaining the calibration ability learned in the first stage.

C. Evaluation Metrics

The paper introduces the ChowScore (adapted from statistical learning) as a unified metric for task accuracy and reflection calibration. It rewards correct commitments ($s_{ref}=1, r=1$) and assigns a fixed credit $\beta$ to honest error flags ($s_{ref}=0$), effectively measuring the agent's ability to know when it has solved the task.

3. Key Results

Experiments were conducted on text-to-SQL tasks across five benchmarks (Spider-Dev, Spider-DK, Spider-Realistic, Spider-Test, Bird-Dev) using Qwen2.5-Coder (3B/7B) and Llama-3.2-3B models.

• Calibration Improvement: RefGRPO drastically reduces the underconfidence rate. In the multi-turn setting with Qwen2.5-Coder-7B, the underconfidence rate dropped from 44.4% (GRPO+) to 7.7%.
• Task Accuracy: Unlike standard RL which often trades calibration for accuracy, RefGRPO improves both. Task accuracy increased from 75.1% to 76.5% for the 7B model.
• Unified Metric: The ChowScore improved significantly (e.g., from 73.0% to 76.5% for Qwen-7B), demonstrating simultaneous gains in solving tasks and knowing when the solution is correct.
• Comparison to Specialists: Compared to specialized 7B SQL models (OmniSQL-7B and SQL-R1-7B), RefGRPO achieved the highest "calibration delta" ($\Delta = \text{Acc}_{ref} - \text{Acc}$), indicating its reflection is genuinely informative rather than just a byproduct of high accuracy.

4. Significance and Implications

The paper claims that RefGRPO transforms the agent into its own verifier, grounded in environment feedback rather than pure self-assessment. This calibrated reflection enables two downstream capabilities:

1. Verifier-Free Self-Improvement: Because the reflection is calibrated, it can serve as a reliable pseudo-reward signal. The authors demonstrate that starting from a RefGRPO checkpoint, further self-improvement training (using $s_{ref}$ as the reward without outcome supervision) yields significant accuracy gains (+2.8 points), whereas standard GRPO checkpoints fail to improve because their pseudo-rewards collapse due to miscalibration.
2. Effective Test-Time Selective Prediction: A calibrated agent can act as its own filter during inference. By committing only to rollouts flagged as correct ($s_{ref}=1$), RefGRPO achieves a higher "selective prediction lift" (accuracy on committed samples) compared to baselines, effectively filtering out errors without needing external verification.

5. Limitations and Future Work

The authors note that their largest trained model is 7B due to compute constraints, and validating the method at larger scales remains a future step. Additionally, the current work focuses on binary reflection scores ($s_{ref} \in \{0, 1\}$), and extending the approach to real-valued confidence scores is suggested as a potential direction.

In summary, RefGRPO offers a simple, annotation-free mechanism to align an agent's self-assessment with reality, solving a structural credit-assignment problem in standard RL and enabling more robust, self-verifying agentic behaviors.