🤖 AI

Reinforced Agent: Inference-Time Feedback for Tool-Calling Agents

This paper introduces "Reinforced Agent," an inference-time framework that employs a specialized reviewer to evaluate tool calls before execution, thereby enabling real-time error correction and demonstrating through new Helpfulness-Harmfulness metrics that separating execution from review allows for systematic performance gains via model selection and prompt optimization without retraining the base agent.

Original authors: Anh Ta, Junjie Zhu, Shahin Shayandeh

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anh Ta, Junjie Zhu, Shahin Shayandeh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, fast robot assistant (the Tool-Calling Agent) whose job is to go out and perform tasks for you, like checking the weather, booking a flight, or setting an alarm. This robot is great, but sometimes it makes mistakes: it might pick the wrong tool, use the wrong units (like Celsius instead of Fahrenheit), or try to do something that doesn't need a tool at all.

Usually, when we find out the robot made a mistake, it's after the fact. We see the error, fix the robot's instructions, and hope it doesn't happen again next time. This is like a teacher grading a test after the student has already left the classroom. The damage is done, and the student can't change their answer on the spot.

This paper introduces a new way to work: The Reinforced Agent.

The Core Idea: The "Editor" in the Room

Instead of waiting until the end, the researchers put a second, specialized robot (the Reviewer Agent) in the room before the first robot does anything.

Think of it like a movie editor sitting next to a director.

The Director (Tool Agent) says, "I'm going to cut this scene!"
The Editor (Reviewer Agent) stops them and says, "Wait! You're cutting the wrong scene. Also, you're using the wrong camera angle. Let's fix that before we hit 'record'."
The Director fixes the plan.
Then, the action happens.

This happens in real-time, right before the tool is actually used. If the plan is good, the Editor says "Go!" and the tool runs. If the plan is bad, the Editor gives feedback, and the Director tries again immediately.

The Big Trade-off: Helpful vs. Harmful

The researchers realized that having an Editor isn't always perfect. Sometimes, the Editor might be too picky and tell the Director to change a good plan, making it worse. Or, the Editor might miss a mistake.

To measure this, they invented two simple scores:

Helpfulness: How often did the Editor catch a real mistake and fix it?
Harmfulness: How often did the Editor mess up a plan that was actually already correct?

They found that the type of "brain" used for the Editor matters a lot. They tested a standard smart model (GPT-4o) and a "reasoning" model (o3-mini) that thinks more carefully.

The Reasoning Model was like a very careful, logical editor. It caught many mistakes without ruining good plans. For every 3 mistakes it fixed, it only made 1 new mistake (a 3:1 ratio).
The Standard Model was a bit more hasty. It fixed fewer mistakes and accidentally broke more good plans (a 2:1 ratio).

The Results: Better Accuracy, But Slower Speed

When they tested this system on two different types of tasks:

Single-turn tasks (like asking "What's the weather?" once): The system got significantly better at knowing when no tool was needed (improving by 5.5%).
Multi-turn tasks (like a long conversation about booking a trip with many steps): The system improved by 7.1%.

The Catch (Latency):
Because the system has to pause, ask the Editor for an opinion, and then wait for a reply, it takes longer.

For a simple, one-off task, the process became 6 times slower.
For a long, complex conversation, the slowdown was less noticeable (about 2.4 times slower) because the "Editor" time gets spread out over many steps of the conversation.

The Secret Sauce: Auto-Optimization

The researchers also found that writing the Editor's instructions (the "prompt") by hand is hard. They used a system called GEPA to automatically rewrite the Editor's instructions. The system looked at times the Editor failed, figured out why, and wrote a better rulebook. This automatically improved the Editor's performance by another 1.5% to 2.8% without needing to retrain the main robot.

Summary

The paper shows that by adding a "second pair of eyes" that checks the work before it's done, we can make AI agents much more accurate.

Pros: Fewer mistakes, no need to retrain the main AI, and the "Editor" can be upgraded independently.
Cons: It takes more time to get a result.
Best Use: It's perfect for complex, important tasks where accuracy matters more than speed (like booking a flight or managing a database), but it might be too slow for simple, instant questions.

The paper does not claim this works for medical diagnosis or clinical uses; it focuses strictly on improving how AI agents interact with software tools and APIs.

1. Problem Statement

Tool-calling Large Language Models (LLMs) face systematic challenges in selecting the correct tools, constructing accurate parameters, and recognizing when no tool is relevant. Current evaluation methods are largely post-hoc (occurring after execution), meaning errors are identified too late to be corrected in real-time.

Existing solutions fall into two categories, both with significant limitations:

Training-based approaches (e.g., GRPO): Require massive computational resources and slow deployment cycles.
Inference-time self-correction (e.g., Self-Refine, Reflexion): While training-free, they struggle with state recovery. If an agent executes a destructive action (e.g., deleting an alarm), it cannot easily revert the state without an impractically large context window to maintain alternative trajectories.

The core problem is the lack of a mechanism to proactively evaluate and correct tool calls before execution without requiring retraining or complex state management.

2. Methodology: Reinforced Agent Architecture

The authors propose a duo-agent architecture that separates execution from review. This system introduces a specialized Reviewer Agent that operates within the inference loop.

Core Workflow

Provisional Call: The primary Tool-Calling Agent generates a tool call.
Review Loop: Before the call is executed, the Reviewer Agent evaluates it.
- If errors are detected, the Reviewer provides feedback (e.g., "Use Fahrenheit for US cities").
- The Tool-Calling Agent revises the call based on this feedback.
- This loop repeats up to $N$ times or until the call is approved.
Execution: Only approved tool calls are sent to the execution environment.

Feedback Mechanisms Evaluated

The paper compares three collaboration strategies:

Progressive Feedback ( $r_N$ ): Iterative refinement where the reviewer provides system message feedback, and the agent revises.
Best-of-N Selection ( $s_N$ ): The agent generates $N$ candidates; a selector picks the best one.
Best-of-N Grading ( $g_N$ ): The agent generates $N$ candidates; a grader assigns numeric scores and rationales to select the best.

Automated Prompt Optimization (APO)

To address the labor-intensive nature of manual prompt engineering, the authors utilize GEPA (Genetic-Pareto prompt evolution). This system automatically refines the Reviewer's prompt by observing cases where the reviewer made incorrect judgments, iterating to produce more robust instructions without changing the base agent's prompt.

3. Key Contributions & Metrics

The paper introduces a novel framework for measuring the trade-offs of adding a reviewer agent.

Helpfulness-Harmfulness Metrics

To quantify whether a reviewer provides net positive value, the authors define:

Helpfulness: The percentage of base agent errors that the reviewer successfully corrects.
Harmfulness: The percentage of correct base agent responses that the reviewer incorrectly flags and degrades.
Benefit-to-Risk Ratio: The ratio of Helpfulness to Harmfulness.

This metric reveals that a reviewer is only valuable if it corrects significantly more errors than it introduces.

Model Selection

The study compares standard language models (GPT-4o) against reasoning models (o3-mini, GPT-5 mini). The hypothesis is that reasoning models, with their systematic verification capabilities, will exhibit higher helpfulness and lower harmfulness.

4. Experimental Results

The approach was evaluated on two benchmarks: BFCL (single-turn, stateless) and $\tau^2$ -Bench (multi-turn, stateful with domain policies).

Performance Gains

BFCL (Single-Turn):
- Achieved +5.5% improvement in Irrelevance Detection (84.9% $\to$ 90.4%).
- Achieved +1.6% improvement in the Relevance Suite.
- Reasoning Models: The o3-mini reviewer achieved a 3.1:1 benefit-to-risk ratio (36.8% helpfulness vs. 11.7% harmfulness), outperforming GPT-4o (2.1:1).
$\tau^2$ -Bench (Multi-Turn):
- Achieved +7.1% average improvement across domains (Airline, Retail, Telecom) (48.7% $\to$ 55.8%).
- Progressive Feedback significantly outperformed Best-of-N selection strategies in stateful scenarios.
- The system effectively reduced Policy Constraint Violations (e.g., checking seat availability before booking) by 13%.

Impact of Optimization

GEPA Optimization: Automated prompt optimization provided an additional +1.5% to +2.8% gain on BFCL, particularly in complex parallel tool-calling scenarios where manual prompts failed to catch subtle mismatches.

Latency Analysis

Single-Turn: Latency increased by 6.2 $\times$ (1.27s $\to$ 7.87s) because the reviewer overhead dominates a single inference call.
Multi-Turn: Latency increased by 2.4 $\times$ (158.7s $\to$ 384.3s). The overhead is amortized across ~40 turns per episode, making it more viable for complex workflows.

5. Significance and Conclusion

This work demonstrates that separating execution and review is a highly effective strategy for improving tool-calling agents without retraining.

Proactive Error Mitigation: By reviewing calls before execution, the system prevents destructive state changes, solving the state recovery problem inherent in self-correction.
Modular Improvement: Organizations can improve agent reliability by swapping in better reviewer models or optimizing reviewer prompts (via GEPA) without touching the base agent's training or architecture.
Reasoning Models as Reviewers: The study provides empirical evidence that reasoning models (like o3-mini) are superior reviewers due to their ability to systematically verify constraints, resulting in a high benefit-to-risk ratio.
Deployment Viability: While latency increases, the trade-off is acceptable for accuracy-critical, multi-turn workflows. Future work suggests distilling the reviewer into smaller, faster models for real-time deployment.

In summary, "Reinforced Agent" shifts the paradigm from post-hoc recovery to proactive evaluation, offering a practical, training-free path to more reliable autonomous agents.