🤖 AI

Towards a Science of AI Agent Reliability

This paper proposes a framework of twelve metrics across four dimensions—consistency, robustness, predictability, and safety—to holistically evaluate AI agent reliability, revealing that recent capability improvements have yielded only marginal gains in actual operational reliability.

Original authors: Stephan Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala, Arvind Narayanan

Published 2026-06-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Stephan Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala, Arvind Narayanan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "A+ Student" Who Can’t Be Trusted

Imagine you have a brilliant student who gets an A+ on every test. You assume they are perfect. But then, you ask them to do a simple chore, like making a sandwich.

Run 1: They make a perfect sandwich.
Run 2: They forget the bread.
Run 3: They use ketchup instead of mustard.
Run 4: They accidentally delete the recipe book.

Even though the student is "smart" (high accuracy), they are unreliable. If you can’t predict what they will do next, you can’t trust them to run your kitchen alone.

This paper argues that we are judging AI agents (software that acts on its own, like coding assistants or customer service bots) only by their "test scores" (accuracy). But in the real world, reliability matters more than raw intelligence. The authors found that while AI has gotten much smarter over the last two years, it hasn’t gotten much more reliable.

The Four Pillars of Reliability

To fix this, the authors borrowed ideas from industries where mistakes are dangerous, like aviation and nuclear power. They broke "reliability" down into four specific dimensions. Think of these as four different ways to judge if a machine is trustworthy:

1. Consistency (The "Same Result Every Time" Test)

The Analogy: If you press the "brew" button on your coffee maker, it should always make coffee. It shouldn’t sometimes make coffee, sometimes make tea, and sometimes explode.
The Paper’s Finding: AI agents are surprisingly inconsistent. If you ask an AI to solve the same problem five times, it might succeed three times and fail twice, even with the exact same instructions.
Why it matters: If an AI code-writer fixes a bug one day but breaks it the next day when you ask it to do the same thing, you can’t rely on it.

2. Robustness (The "Bad Day" Test)

The Analogy: A robust car engine should still run if it’s raining, if the road is bumpy, or if the gas is slightly different. It shouldn’t stall just because a leaf blew into the air intake.
The Paper’s Finding: AI agents are fragile. If you slightly rephrase a question (e.g., changing "Cancel my order" to "I want to end my subscription"), the AI might fail. If a tool it uses glitches for a second, the AI might give up entirely instead of trying again.
Why it matters: In the real world, things are messy. If an AI can’t handle small changes in how you talk to it or minor technical glitches, it will fail in production.

3. Predictability (The "Honest Confidence" Test)

The Analogy: Imagine a weather forecaster who says, "I am 90% sure it will rain." If it rains 90% of the time they say that, they are predictable. If they say "90% sure" but it only rains 50% of the time, they are overconfident and unpredictable.
The Paper’s Finding: AI agents are often bad at knowing when they are about to fail. They might say, "I’m very confident I solved this," when they actually got it wrong.
Why it matters: If you can’t trust the AI’s confidence score, you don’t know when to double-check its work. You either have to check everything (which is slow) or trust it blindly (which is dangerous).

4. Safety (The "Don’t Burn the House Down" Test)

The Analogy: A chef might make a bad meal (a failure), but a safe chef won’t accidentally set the kitchen on fire (a catastrophic failure).
The Paper’s Finding: The authors measured how often AI agents broke rules (like deleting files they shouldn’t or exposing private data). While recent models are better at following rules, rare "high-severity" mistakes still happen.
Why it matters: A small error is annoying; a catastrophic error is costly. Reliability isn’t just about being right; it’s about bounding how bad the mistakes are when they happen.

The Big Surprise: Smarter ≠ More Reliable

The authors tested 15 different AI models from OpenAI, Google, and Anthropic. They looked at data from 2024 to 2026.

Accuracy went up: The models got much better at solving tasks.
Reliability stayed flat: The models did not get significantly better at being consistent, robust, or predictable.

The Metaphor: It’s like upgrading from a bicycle to a Ferrari. The Ferrari is much faster (higher accuracy), but if the steering wheel is loose and the brakes are unpredictable (low reliability), you still can’t trust it to drive your children to school.

What Should We Do?

The paper suggests four main recommendations for the industry:

Stop using single tests: Don’t just ask the AI to do a task once. Ask it to do it 10 times. Change the wording. Break the tools on purpose. See if it still works.
Design for reliability, not just smarts: When building AI, engineers should explicitly try to make the AI more consistent and robust, not just more accurate.
Use reliability scores for hiring: Just like a pilot needs a license to fly, an AI agent should need a "reliability certificate" before it’s allowed to work autonomously in a company.
Match the risk to the role:
- Assistant Mode: If a human is checking the AI’s work (like a coding assistant), lower reliability is okay because the human is the safety net.
- Autonomous Mode: If the AI is acting alone (like a customer service bot that processes refunds), it needs very high reliability because there is no human to catch its mistakes.

In Summary

This paper is a call to action. It says we are too focused on making AI smarter and not enough on making it trustworthy. We need new ways to measure AI that look at consistency, robustness, predictability, and safety—not just whether it got the right answer one time. Until we do that, AI agents will remain like brilliant but erratic interns: impressive in the lab, but risky in the real world.

Technical Summary: Towards a Science of AI Agent Reliability

Problem Statement
The paper addresses a critical discrepancy in the evaluation of AI agents: while accuracy scores on standard benchmarks have risen rapidly, agents frequently fail in real-world deployments. The authors argue that the dominant paradigm of evaluating agents via mean task success rates (accuracy) is insufficient because it obscures critical behavioral properties. Accuracy alone cannot distinguish between an agent that fails predictably on specific tasks and one that fails randomly, nor can it differentiate between benign formatting errors and catastrophic actions (e.g., deleting production databases). This evaluation gap contrasts sharply with safety-critical engineering domains (aviation, nuclear power, automotive), where reliability is treated as a multi-dimensional property involving consistency, robustness, predictability, and bounded failure severity.

Methodology
The authors propose a holistic framework for evaluating AI agent reliability, grounded in safety-critical engineering principles. They decompose reliability into four independent dimensions, each operationalized with specific metrics:

Consistency ( $R_{Con}$ ): Measures repeatable behavior under identical conditions.
- Outcome Consistency ( $C_{out}$ ): Variance in success/failure across repeated runs.
- Trajectory Consistency: Distributional ( $C^d_{traj}$ ) and sequential ( $C^s_{traj}$ ) similarity of action paths.
- Resource Consistency ( $C_{res}$ ): Variability in computational/monetary costs.
Robustness ( $R_{Rob}$ ): Measures stability under perturbations.
- Fault Robustness ( $R_{fault}$ ): Resilience to infrastructure failures (e.g., API timeouts).
- Environment Robustness ( $R_{env}$ ): Stability against semantic-preserving changes in data formats or tool interfaces.
- Prompt Robustness ( $R_{prompt}$ ): Invariance to semantically equivalent instruction paraphrases.
Predictability ( $R_{Pred}$ ): Measures the alignment between expressed confidence and actual performance.
- Calibration ( $P_{cal}$ ): Alignment of confidence scores with empirical success rates (Expected Calibration Error).
- Discrimination ( $P_{AUROC}$ ): Ability of confidence scores to separate successes from failures (AUC-ROC).
- Brier Score ( $P_{brier}$ ): A proper scoring rule jointly measuring calibration and discrimination.
Safety ( $R_{Saf}$ ): Measures bounded harm when failures occur.
- Compliance ( $S_{comp}$ ): Adherence to predefined constraints (e.g., no unauthorized actions).
- Harm Severity ( $S_{harm}$ ): Severity of consequences for tasks that violate constraints.

The authors evaluate 15 models from OpenAI, Google, and Anthropic (released between early 2024 and mid-2026) across two complementary benchmarks:

GAIA: A general assistant benchmark involving web browsing, file manipulation, and multi-step reasoning.
$\tau$ -bench: A customer service simulation benchmark involving multi-turn conversations and consequential database actions.

The evaluation protocol involves multi-run executions ( $K=5$ ) with temperature set to zero to isolate non-sampling stochasticity, along with systematic perturbations for robustness and post-hoc self-assessment for confidence estimation.

Key Contributions

A Formal Taxonomy and Metric Suite: The paper translates qualitative safety-critical principles into 12 computable metrics that evaluate reliability independently of raw task success.
A Comprehensive Reliability Profile: The authors provide a detailed mapping of where state-of-the-art models succeed and fail, identifying consistency and predictability as areas requiring immediate research focus.

Results

Reliability Gains Lag Capability: Despite 24 months of model development and significant improvements in accuracy, overall reliability has shown minimal improvement. The correlation between accuracy and reliability is weak, particularly on the open-ended GAIA benchmark.
Dimension-Specific Findings:
- Consistency: Outcome consistency remains low across all models. Agents exhibit a "what but not when" pattern, reliably selecting similar action types but varying in execution order. Resource consistency is particularly poor on GAIA.
- Robustness: Fault and environment robustness show ceiling effects (high scores), but prompt robustness varies significantly across models. Agents remain vulnerable to superficial instruction paraphrasing.
- Predictability: Calibration has improved in recent models (particularly Claude), but discrimination has stagnated or worsened on GAIA. This indicates that while models are better at estimating their average success rate, they are not better at identifying which specific tasks they will fail.
- Safety: Recent frontier models exhibit lower violation rates. However, financial accuracy violations remain the most common failure mode in $\tau$ -bench.
Model Type Analysis: Reliability does not scale uniformly with capability. While calibration, robustness, and safety generally improve with model size, consistency often exhibits an inverse pattern, with smaller models sometimes achieving higher consistency. Reasoning models are generally more reliable than non-reasoning counterparts, but their reliability gains lag behind their accuracy improvements.

Significance and Recommendations
The paper argues that treating reliability as an independent axis of progress is essential for the safe deployment of AI agents. The authors propose four recommendations:

Dynamic Benchmarks: Evaluation must move beyond single-run accuracy to include multi-run protocols and systematic perturbations that mirror real-world distribution shifts.
Architecture Design: Agent architectures should be explicitly optimized for reliability dimensions, not just capability.
Deployment Governance: Reliability metrics should inform deployment decisions, change management, and compliance, analogous to certification requirements in safety-critical industries.
Autonomy Scaling: Reliability requirements should scale with the degree of agent autonomy; agents operating without human buffers require stricter reliability thresholds than those used in augmentation settings.

The authors conclude that the core shift in perspective is from asking "How often does the agent succeed?" to asking "How predictably, consistently, robustly, and safely does it behave?" They view their framework as a starting point for establishing reliability as a standard evaluation dimension, complementing rather than replacing careful deployment practices.