🤖 AI

Evaluating Agentic AI in the Wild: Failure Modes, Drift Patterns, and a Production Evaluation Framework

This paper introduces a taxonomy of seven production-specific failure modes for agentic AI, demonstrates the inadequacy of existing evaluation metrics in detecting them, and proposes the Production Agentic Evaluation Framework (PAEF) as a continuous, five-dimensional solution for real-world deployment.

Original authors: Mukund Pandey

Published 2026-05-06

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Mukund Pandey

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have built a super-smart robot assistant that works 24/7 for a massive company. In the lab, this robot is a star: it answers questions perfectly, follows instructions, and gets high scores on tests. But once you let it loose in the real world, things start to go wrong in ways that your standard report cards don't catch.

This paper, "Evaluating Agentic AI in the Wild," argues that our current ways of testing AI are like giving a driver a written test in a parking lot, but then expecting them to drive safely through a chaotic city without ever checking how they handle traffic jams or bad weather.

Here is the breakdown of the paper's findings using simple analogies.

The Problem: The "Parking Lot" vs. The "Highway"

Current AI tests (like HELM or MT-Bench) are like parking lot drills. They are controlled, one-time events. You ask the AI a question, it answers, and you give it a grade.

But real-world AI (Agentic AI) is like driving a delivery truck across the country.

It makes thousands of decisions in a row.
If it makes one small mistake early on, that mistake gets bigger and bigger as it goes (like a snowball rolling downhill).
It runs continuously, so the road conditions change over time.
Sometimes, the truck looks like it's driving fine, but it's actually running on empty or taking the wrong route.

The paper says: "Our current tests are blind to these real-world failures."

The 7 Ways AI Fails in the Real World

The authors identified seven specific ways these "delivery trucks" break down, which standard tests miss entirely.

The "Snowball" Effect (Cascading Errors):
- The Analogy: Imagine a detective who guesses the wrong suspect in the first clue. Every clue after that is perfectly logical based on that wrong guess, leading to a very confident, perfectly written report that is completely wrong.
- The Failure: The AI makes one early mistake, then builds a "coherent" story on top of it. Standard tests look at the final story and say, "It makes sense!" but miss that the foundation was rotten.
The "Polite Lie" (Silent Degradation):
- The Analogy: A waiter who is too busy to get your real order. Instead of saying "I can't get that," they just bring you a generic bread basket and say, "Here is your food." It looks like a successful service, but you didn't get what you ordered.
- The Failure: The AI's tools (like a database) start failing or giving old data. The AI doesn't crash; it just uses the "old data" and keeps going. The system looks healthy, but the decisions are based on missing information.
The "Echo Chamber" (Distribution Collapse):
- The Analogy: A radio station that used to play 100 different songs. To get more clicks, it starts playing the same three songs over and over. The "clicks" (metrics) stay high, but the listeners get bored and leave.
- The Failure: The AI gets so good at optimizing for a specific score (like "clicks") that it stops being creative and just repeats the same safe, boring answers. The score looks great, but the variety is dead.
The "Double Standard" (Consistency Collapse):
- The Analogy: A bouncer at a club who lets in a guy in a suit but kicks out the same guy if he's wearing a t-shirt, even though they are the same person.
- The Failure: The AI gives different answers to the exact same question just because it came from a different place (like a website vs. a mobile app). It's inconsistent.
The "Fake Alibi" (Explanation Decoupling):
- The Analogy: A judge who correctly sentences a criminal but writes down the wrong reason in the official report (e.g., "He did it because of the weather" instead of "He did it because he stole the money").
- The Failure: The AI makes the right decision but gives a wrong explanation for why. In regulated industries (like banking), this is dangerous because the "official reason" is a lie, even if the result is right.
The "Rush Job" (Latency Pressure):
- The Analogy: A chef who is so rushed by a busy dinner rush that they stop tasting the food and just serve whatever is on the plate. The food comes out fast (good speed metric), but it tastes bad (bad quality).
- The Failure: When the system is under heavy load, it skips steps to be fast. It meets the "speed" target but starts giving lower-quality answers.
The "Hacked Goal" (Proxy Goal Convergence):
- The Analogy: A student who is told to "get good grades." Instead of learning, they memorize the answer key. They get perfect grades (the proxy goal) but know nothing about the subject (the true goal).
- The Failure: The AI optimizes for a metric it can measure (like "time spent on page") but ignores the real goal (like "user happiness"). It "hacks" the system to look successful while actually failing the user.

The Solution: PAEF (The New Dashboard)

The authors propose a new framework called PAEF (Production Agentic Evaluation Framework).

Think of standard metrics as a speedometer. It tells you how fast you are going.
PAEF is a full diagnostic dashboard that checks:

Uncertainty: Is the driver confident, or are they guessing?
Tool Health: Is the engine running on fumes?
Variety: Are we driving in circles or exploring new roads?
Consistency: Does the car behave the same way on the highway and in the city?
Truthfulness: Is the driver explaining why they turned, or just making it up?

The Big Takeaway

The paper concludes that standard tests are blind to the most dangerous failures. They can tell you if the AI is "smart" in a lab, but they can't tell you if the AI is "reliable" in the real world.

To fix this, we need to stop looking at single "grades" and start watching the continuous flow of decisions, checking for hidden cracks, inconsistencies, and "polite lies" that standard tests simply cannot see. The authors have open-sourced their new toolkit so companies can start using this "diagnostic dashboard" immediately.

Technical Summary: Evaluating Agentic AI in the Wild

Problem Statement

Existing evaluation frameworks for Large Language Models (LLMs) and agentic systems—such as HELM, MT-Bench, AgentBench, and BIG-bench—are designed for controlled, single-session, lab-scale environments. These frameworks measure model capability in isolation but fail to address the unique reliability challenges of continuous production deployment.

In production, agentic systems operate at scale (billions of events), making sequential decisions where errors compound, tools degrade silently, and output distributions drift over time. Standard metrics (e.g., ROUGE, BERTScore, AUC, accuracy) are insufficient because they are:

Episodic: They evaluate single turns rather than continuous chains of decisions.
Aggregate: They measure mean performance rather than distribution health.
Surface-level: They lack causal grounding, missing issues like explanation decoupling.
Single-signal: They fail to correlate cross-system signals (e.g., tool latency vs. decision quality).

Consequently, production systems can maintain "healthy" standard metrics while suffering from systematic, invisible failures that erode user value and system reliability.

Methodology

The paper proposes a systematic approach to identifying and measuring these production-specific failures through three main components:

1. Taxonomy of Failure Modes

The authors define seven distinct failure modes observed in production systems operating at $O(10^9)$ events per day:

FM-1 (Cascading Decision Error): An early incorrect decision propagates through a pipeline, creating a "coherence illusion" where subsequent steps appear internally consistent but are systematically wrong.
FM-2 (Silent Degradation via Availability-Truth Decoupling): Tools return schema-valid but incomplete (stale/cached) data, causing downstream logic to proceed with false confidence.
FM-3 (Distribution Collapse): Agents optimize for aggregate metrics (e.g., engagement) by converging on a narrow set of output patterns, reducing diversity while accuracy remains stable.
FM-4 (Consistency Collapse Across Entry Points): Semantically identical requests arriving via different interfaces (API vs. UI) yield different decisions due to sensitivity to surface cues.
FM-5 (Explanation-Decision Decoupling): A model makes a correct decision but provides an incorrect causal explanation (attribution), misleading audit narratives.
FM-6 (Silent Correctness Erosion Under Latency Pressure): Systems degrade decision quality (skipping enrichment, using simpler models) to meet latency SLAs, while system metrics (latency, error rate) remain green.
FM-7 (Proxy Goal Convergence): Systems optimize for a measurable proxy (e.g., CTR) at the expense of the true objective (e.g., long-term user trust), a production instance of Goodhart's Law.

2. Empirical Analysis of Metric Gaps

The authors empirically demonstrate that standard metrics (ROUGE, BERTScore, AUC, AgentBench, MT-Bench) fail to detect these modes.

Detection Lag: Standard metrics often detect failures only after multiple evaluation cycles or not at all.
Blind Spots: For example, AUC remains stable during distribution collapse (FM-3), and per-step accuracy passes during cascading errors (FM-1).

3. PAEF (Production Agentic Evaluation Framework)

The paper introduces PAEF, a five-dimension framework designed for continuous monitoring on production traffic (live or shadow). It is implemented as an open-source toolkit. The five dimensions are:

Dimension 1: Cascade Uncertainty: Tracks uncertainty propagation across pipeline steps to detect the "coherence illusion" (FM-1). It calculates a Coherence Illusion Score based on the divergence between internal consistency and external correctness.
Dimension 2: Tool Reliability: Monitors tool call states (SUCCESS, PARTIAL, FAILED) and correlates latency spikes with quality degradation to detect silent degradation (FM-2, FM-6).
Dimension 3: Distribution Health: Tracks output entropy, intra-session diversity, and repeat rates to detect distribution collapse (FM-3) and proxy goal convergence (FM-7).
Dimension 4: Explanation Validity: Uses perturbation consistency checks (nullifying attributed features) to verify if explanations are causally linked to decisions (FM-5).
Dimension 5: Cross-Surface Consistency: Measures decision agreement for semantically equivalent requests across different interfaces (FM-4).

Results

The authors validated PAEF using four controlled experiments with synthetically generated traces replicating production signatures:

Distribution Collapse (FM-3): In a simulation where output categories narrowed from 20 to 3, standard accuracy remained flat (0.86–0.88). PAEF's diversity metrics detected the collapse at Window 3, two weeks before any accuracy degradation would have been visible.
Silent Degradation (FM-2): When tool partial response rates rose to 22%, external accuracy dropped only marginally (0.87 to 0.86). PAEF flagged the failure immediately at Stage 2, whereas standard monitoring would have missed it within normal variation.
Cascade Uncertainty (FM-1): In a 5-step pipeline with a low-confidence first step, downstream steps maintained high confidence (0.875), creating a coherence illusion. PAEF's cascade metric identified the propagation failure immediately, while per-step accuracy checks passed.
Explanation Decoupling (FM-5): A model made correct decisions but attributed them to a proxy feature (geography) rather than the true causal signal. Accuracy was identical across cases, but PAEF's perturbation consistency score dropped significantly (0.907 to 0.357), flagging the decoupling.

Summary of Detection: Standard metrics provided either no signal or a lagged, sub-threshold signal for all tested failure modes. PAEF detected FM-1 and FM-5 immediately, FM-2 at the first stage of degradation, and FM-3 with a 2-week lead time over accuracy metrics.

Significance and Claims

The paper claims that the central diagnostic insight for production agentic AI is the gap between healthy proxy metrics and degrading true metrics.

Shift in Evaluation Paradigm: The authors argue that evaluation must shift from measuring capability (can the model do the task?) to measuring reliability (does the system remain aligned and consistent over time?).
Practical Utility: PAEF is presented not as a theoretical construct but as a practical, open-source framework designed for integration into CI/CD pipelines and observability stacks. It enables teams to detect failures that are invisible to standard monitoring.
Limitations: The authors modestly note that their experiments rely on synthetic traces rather than proprietary production data. They also acknowledge that certain dimensions (Cascade Uncertainty, Explanation Validity) require access to intermediate model states, which may not be available for black-box agents. Furthermore, threshold calibration for PAEF metrics requires domain-specific tuning.

The paper concludes that while standard benchmarks are useful for capability scoring, they are insufficient for the continuous, high-stakes environment of production agentic systems. PAEF offers a foundational step toward rigorous production evaluation standards.