💬 NLP

Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench

This paper introduces AgentProp-Bench, a human-validated benchmark revealing that automated judges for tool-using LLM agents often perform at chance levels, characterizing how parameter injection errors propagate to final answers, and demonstrating that runtime mitigation strategies can significantly reduce hallucinations in specific models.

Original authors: Bhaskar Gurram

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bhaskar Gurram

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have hired a team of very smart, but slightly clumsy, robot assistants. Their job is to solve problems by using a set of tools, like a calendar app, a weather forecaster, or a medical database. You ask them, "Book a meeting for tomorrow," and they are supposed to type the right commands into the computer to make it happen.

This paper is about a new way to test these robots to see how often they mess up, why they mess up, and if we can build a safety net to catch them before they fail.

Here is the breakdown of their findings, using some everyday analogies.

1. The Problem: The "Bad Judge"

For a long time, people tested these robot assistants by looking at their final answer and checking if it contained the right keywords.

The Old Way: If the robot was supposed to say "The weather in London is sunny," and it said "The weather in London is very sunny," the old test would say, "Great job! You got it right."
The Reality: The researchers found this method is basically flipping a coin. It's so unreliable that it agrees with human experts only 5% of the time (which is worse than random guessing). It's like a teacher grading a math test by only checking if the student wrote the number "5" anywhere on the page, ignoring whether the math was actually correct.

The Fix: They built a "Super-Judge Panel" made of three different AI models voting on the answers. This panel is much better (about 43% agreement with humans), but it's still not perfect. It tends to be a "strict teacher," often marking answers as wrong even when they are actually okay.

2. The Experiment: The "Sabotage Test"

To see how errors spread, the researchers didn't just wait for the robots to make mistakes naturally. They sabotaged them on purpose.

The Analogy: Imagine a chef (the robot) is making a soup. The researchers secretly swapped the salt for sugar before the chef started cooking.
The Question: Does the chef notice the sugar and fix it? Or does the chef keep cooking, serve the sweet soup, and tell you it's delicious?

They found that when they swapped a critical piece of information (like changing "London" to "Manchester" in a weather query):

62% of the time, the robot didn't notice the swap. It used the wrong info, did the rest of the work, and gave you a confidently wrong answer.
The "Domino Effect": Once the robot made that first small mistake, it rarely recovered. It was like a domino falling; once the first one tipped, the whole line crashed.

3. Two Different Skills: The "Bouncer" vs. The "Detective"

The researchers discovered that robots have two completely different skills when it comes to errors, and being good at one doesn't mean you're good at the other.

Skill A: The Bouncer (Rejection): This is the robot's ability to say, "Wait, this tool call looks weird. I'm not sending it."
- Example: One robot (Gemini) was a super-tight bouncer. It caught 95% of the bad inputs before they even left the building.
Skill B: The Detective (Recovery): This is the robot's ability to realize, "Oh no, I already sent the wrong request. Can I fix the final answer anyway?"
- Example: Another robot (GPT-4o-mini) was a terrible detective. Even if it let a bad request through, it almost never figured out how to fix the final result.

The Big Surprise: Being a great Bouncer doesn't make you a good Detective. You can have a robot that is excellent at stopping errors but terrible at fixing them, or vice versa. This means we can't just say a robot is "smart"; we have to test these two skills separately.

4. The Safety Net: The "Runtime Interceptor"

The team built a "Safety Net" (called an Interceptor) that watches the robot while it works. It's like a security guard standing next to the chef, reading the recipe and the ingredients as they go.

How it works: If the robot tries to use a tool with a weird parameter, or if the robot's internal thoughts sound confused ("I'm not sure," "This looks wrong"), the guard hits the brakes.
The Results:
- For the robot that was a "bad detective" (GPT-4o-mini), the Safety Net caught 23% more errors. It saved the day!
- For the robot that was already a "super bouncer" (Gemini), the Safety Net did nothing. Why? Because the robot was already stopping the errors before the guard even had a chance to look. The guard was redundant.

5. The Takeaway

This paper teaches us three main things for the future of AI:

Stop using keyword checks: They are useless for judging complex robot tasks. We need smarter, human-verified ways to grade them.
Errors spread fast: If a robot gets a tiny piece of info wrong, it usually ruins the whole answer. We need to catch errors early.
One size does not fit all: Some robots are good at stopping errors, others are good at fixing them. And safety nets only work if the robot actually needs them.

In short: We can't just trust robots to do their job. We need to test them with "sabotage," grade them with human-like judges, and build safety nets that are tailored to the specific robot's weaknesses.

1. Problem Statement

Current evaluation of tool-using Large Language Model (LLM) agents suffers from three critical gaps:

Unreliable Evaluation Metrics: Most benchmarks rely on substring or keyword-matching heuristics to judge agent outputs. The authors argue these are unreliable for agents, which often produce verbose or paraphrased answers, leading to incorrect correctness assessments.
Lack of Granularity: Existing benchmarks (e.g., $\tau$ -bench, AgentBench) report only a single end-to-end correctness score. They fail to localize where a failure occurs (e.g., parameter generation vs. tool execution vs. final synthesis) or how errors propagate through the pipeline.
Unvalidated Mitigation: Runtime mitigation strategies are often evaluated against historical baselines rather than concurrent controls, making it difficult to isolate the true effectiveness of interventions like hallucination interceptors.

2. Methodology: AgentProp-Bench

The authors introduce AgentProp-Bench, a comprehensive benchmark designed to address these gaps through controlled experimentation and validated evaluation.

Dataset and Design

Scale: 2,000 core tasks across four domains (Calendar, Weather, Medical, Knowledge) plus a held-out Retail domain, totaling 2,300 traces.
Models: Evaluated across 9 production LLMs (including GPT-4o, GPT-4o-mini, Gemini-2.0/2.5-Flash, DeepSeek-V3, etc.).
Controlled Injection (P2 Protocol): Instead of relying on natural errors, the authors use a "semantic-wrong" injection protocol. Before an agent's first tool call, a semantically critical parameter is replaced with a schema-valid but incorrect value (e.g., changing "London" to "Manchester"). This allows for causal measurement of error propagation.
Human Validation: A stratified subset of 100 traces was manually labeled by human annotators to validate the automated judging methods.

Evaluation Framework

The study defines three independent stage indicators to measure error propagation:

$S_1$ (Injection Applied): The agent's planned tool call was modified by the injection.
$S_2$ (Visible Execution): The modified call executed and produced an observable effect (e.g., tool error or altered response).
$S_3$ (Wrong Final Answer): The agent's final output is incorrect.

Judging Protocols:

Heuristic: Substring matching (standard practice).
LLM Ensemble: A majority vote of three LLMs (GPT-4o, Gemini-2.5-Flash, GPT-4o-mini).
Human: Ground truth labels.
Calibration: The ensemble judge showed a conservative bias (under-reporting correctness). The authors derived human-calibrated probabilities to adjust the automated results.

Runtime Mitigation

A three-layer Interceptor was tested under a concurrent control ( $n=600$ tasks per model):

L1 (Schema Validation): Counts parameter/tool errors.
L2 (Thought Monitoring): Scans chain-of-thought for uncertainty keywords.
L3 (Output Consistency): Checks for numerical claims unsupported by tool observations.

3. Key Contributions

Judge Reliability Validation: Demonstrated that substring-based evaluation agrees with human annotation at chance levels ( $\kappa = 0.049$ ), while a three-LLM ensemble achieves moderate agreement ( $\kappa = 0.432$ ).
Stage-Level Propagation Measurement: Quantified how parameter-level injections propagate to final errors, revealing that the probability of an injected error causing a wrong answer is $\approx 0.62$ (human-calibrated).
Rejection-Recovery Decomposition: Proved that a model's ability to reject bad parameters (input filtering) and recover from bad observations (reasoning correction) are statistically independent capabilities ( $\rho = 0.126, p=0.747$ ).
Concurrent Control Mitigation: Evaluated a runtime interceptor against a concurrent no-interceptor control, showing significant reduction in hallucinations for specific models.
Open Release: All code, data, traces, and human labels are publicly available.

4. Key Results

Judge Reliability

Heuristic Failure: Substring matching has a Cohen's $\kappa$ of 0.049 against humans, effectively random.
Ensemble Success: The 3-LLM ensemble reached $\kappa = 0.432$ (moderate agreement). However, it exhibited a conservative bias, marking only 25% of traces as correct compared to 38% by humans. This bias underestimates true correctness by ~13 percentage points.

Error Propagation

Propagation Rate: Under validated evaluation, a parameter injection propagates to a wrong final answer with a probability of ~0.62 (range 0.46–0.73 across models).
Model Variance: Propagation rates vary significantly. For example, DeepSeek-V3 had a lower propagation rate ( $r_{2,3} \approx 0.43$ ) compared to GPT-4.1-nano ( $r_{2,3} \approx 0.97$ ).

Rejection vs. Recovery

Orthogonality: Rejection rate (catching bad inputs) and Recovery rate (fixing bad outputs) are uncorrelated.
- Gemini-2.0-Flash: High rejection (95%), moderate recovery (50%).
- GPT-4.1-nano: Moderate rejection (37.5%), near-zero recovery (2.9%).
Implication: Improving schema validation does not automatically improve reasoning robustness; both must be optimized independently.

Runtime Interceptor Effectiveness

GPT-4o-mini: The interceptor reduced hallucination rates from 55.8% to 32.8% (a 23.0 percentage point reduction, $p < 0.001$ ).
Gemini-2.0-Flash: The interceptor showed no significant effect ( $\Delta = -1.3$ pp). This is because Gemini's aggressive parameter rejection (95%) already eliminates the target failure mode before the interceptor can act.

5. Significance and Implications

For Benchmarking: The paper argues that substring heuristics should be deprecated for agent evaluation due to their chance-level agreement with humans. It advocates for LLM-ensemble judges that are explicitly validated against human labels and calibrated for bias.
For Agent Design: Robustness is multi-dimensional. Developers cannot rely on a single "robustness score." They must separately engineer schema validators (to reject bad inputs) and reasoning mechanisms (to recover from corrupted tool outputs).
For Mitigation Strategies: Runtime interceptors are not universally effective. Their utility depends on the underlying model's rejection capabilities. If a model already filters errors well, an interceptor adds little value; if the model is prone to passing errors, the interceptor is highly effective.
Broader Impact: The findings extend beyond agents to Information Retrieval (IR) and RAG systems, where output variability often renders keyword matching unreliable.

In conclusion, AgentProp-Bench provides a rigorous, validated framework for understanding how errors cascade in tool-using agents, demonstrating that current evaluation methods are flawed and that model robustness requires a dual-focus on input rejection and output recovery.