Technical Summary: ReGRPO – Reflection-Augmented Policy Optimization for Tool-Using Agents

1. Problem Statement

Tool-augmented Vision-Language Models (VLMs) aim to solve complex, multi-modal, multi-step tasks by invoking external tools (e.g., OCR, web search, code execution). However, current approaches suffer from two primary limitations regarding robustness and recovery:

1. Limitations of Supervised Fine-Tuning (SFT): Existing SFT methods (e.g., MAT-AGENT) rely heavily on successful trajectories. They offer little to no signal for recovery when an agent deviates from the optimal path or encounters a tool failure (e.g., an empty OCR result due to a slight layout shift). The model learns to follow successful traces but lacks the capability to diagnose and repair local errors.
2. Limitations of Standard Reinforcement Learning (RL): Standard RL approaches often provide only sparse, trajectory-level rewards (success/failure). In long-horizon tasks, a scalar failure reward does not identify which specific step failed or how to correct it. This ambiguity leads to slow and unstable learning, as the agent cannot distinguish between a wrong tool choice and a wrong argument.

The core challenge is to enable tool-using agents to not only execute tasks but also to diagnose failures, generate structured reflections, and execute corrective actions within a single inference pass without relying on expensive external verifiers or trial-and-error at test time.

2. Methodology: ReGRPO

The authors propose ReGRPO (Reflection-augmented Group Relative Policy Optimization), a framework that treats diagnostic reflection as a learnable variable rather than a fixed prompting heuristic. The methodology consists of three main components:

A. Structured Reflective Data Engine

To address the lack of recovery data in SFT, the authors introduce a data engine that converts tool-execution failures into grounded supervision.

• Failure Induction: Starting from ground-truth trajectories, the system synthesizes "near-miss" actions by perturbing tool choices or arguments (e.g., shifting a bounding box). These are executed in a sandbox to generate real, grounded failure observations (e.g., empty OCR, API errors).
• Reflection-of-Thought (RoT) Triplets: A teacher VLM (default: GPT-4o) analyzes the failure context and generates a structured triplet:
  • ErrorType: Categorical diagnosis (e.g., GroundingDrift, ToolMismatch).
  • Evidence: Explicit reference to the visual or textual observation triggering the error (e.g., "OCR output for region #box1 is empty").
  • FixPlan: A concrete strategy to correct the error (e.g., "Expand the bounding box").
• Supervision: These triplets are paired with the corrected action to form a "Failure → Reflection → Correction" trajectory, used for warm-start SFT.

B. Reflection-Augmented GRPO Training

ReGRPO extends the Group Relative Policy Optimization (GRPO) algorithm to jointly optimize reflection tokens and corrective actions.

• Local Trajectory Formulation: Instead of scoring atomic actions, the policy samples a group of $K$ candidate local trajectories. If an initial action $a^{(0)}$ fails, the policy generates a reflection $z$ and a corrective action $a^{(1)}$.
• Composite Reward Function: The reward $R(\tau)$ balances task success, execution efficiency, and optional verification:
  $$R(\tau) = \lambda_{exec} \mathbb{1}_{\{success\}} - \eta C(\tau) + \lambda_{val} V(x_i, \tau)$$
  • Reflection Cost ($C(\tau)$): Penalizes the token length of the reflection $z$, forcing the agent to reflect only when necessary.
  • Verifier Value ($V$): An optional, training-only signal computed deterministically from metadata (no LLM calls during RL) to stabilize learning.
• Optimization: The policy is updated using group-relative advantages. By comparing recovered trajectories against unrecovered failures within the same group, the model learns which reflections are useful and how to generate them efficiently.

C. Zero-Verifier Inference Stage

At deployment, ReGRPO operates in a single-path, zero-verifier mode to ensure efficiency.

• Deterministic Trigger: A lightweight gate $g_i$ decides whether to open a local reflection-correction block. The trigger activates if:
  1. A tool error is detected.
  2. The observation is empty.
  3. The policy's confidence in the action ($u_i$) falls below an adaptive threshold ($\kappa_i$).
• Recovery: If triggered, the agent executes a single local block: $a^{(0)} \to o^{(0)} \to z \to a^{(1)} \to o^{(1)}$. This allows for self-correction without external verifier calls or search.

3. Key Contributions

The paper makes three primary contributions:

1. Structured Reflective Data Engine: A pipeline that converts raw tool-execution failures into grounded "Error-Reflection-Correction" supervision, enabling agents to learn recovery strategies from the outset.
2. ReGRPO Algorithm: A novel optimization framework that integrates structured reflection into the GRPO objective. It jointly optimizes reflection generation and corrective actions, treating reflection as a decision variable rather than a static prompt.
3. State-of-the-Art Performance: Under identical backbone (Qwen2-VL-7B) and tool suite settings, ReGRPO achieves the best results among compared open-source controllers on the GTA and GAIA benchmarks, outperforming strong baselines like MAT-AGENT and SPORT.

4. Experimental Results

The authors evaluated ReGRPO on two multimodal benchmarks:

• GTA (229 tasks): Focuses on visual perception and local operations (e.g., reading receipts).
• GAIA (446 tasks): Focuses on document understanding, web navigation, and logical reasoning across PDFs, PPTX, and XLSX.

Key Findings:

• Performance: The default verifier-free ReGRPO ($\lambda_{val}=0$) achieved 67.66% Answer Accuracy on GTA and 23.35% on GAIA. This represents a significant improvement over the SPORT baseline (+7.40% on GTA, +2.74% on GAIA).
• Ablation Studies:
  • RoT SFT: Adding structured reflection data for warm-start SFT improved performance over standard SFT, demonstrating the value of explicit error diagnosis.
  • Structured vs. Free-form: Structured reflection (schema-constrained) outperformed free-form reflection, indicating that the ErrorType-Evidence-FixPlan schema is crucial for effective learning.
  • Verifier Independence: While adding an optional teacher-derived verifier reward provided marginal gains, the core ReGRPO pipeline remained highly effective without it, validating the zero-verifier deployment strategy.

5. Significance and Claims

The paper claims that ReGRPO addresses a critical gap in tool-using agents: the ability to recover from failures without relying on brittle supervised traces or expensive external verification.

• Internalizing Correction: Unlike previous methods that treat reflection as a test-time heuristic, ReGRPO internalizes the correction loop into the policy parameters. The model learns when to reflect and how to generate grounded diagnostics that lead to successful recovery.
• Efficiency: By using a deterministic trigger and a single-path execution strategy, ReGRPO maintains low latency and deployment efficiency, avoiding the computational overhead of external verifiers or multi-path search.
• Generalizability: The framework is designed to be model-agnostic regarding the backbone (demonstrated on Qwen2-VL-7B) and tool suite, offering a scalable approach for training robust, self-correcting multimodal agents.

The authors conclude that structured reflection, when optimized via group-relative policy optimization, provides a robust mechanism for handling the distribution shifts and tool failures inherent in real-world multi-step tasks.