💻 computer science

Stop Shipping AI Agents on Faith: Capability Is Not Production Readiness

This paper introduces the ProofAgent Index (PAI), a governance readiness framework that shifts AI agent deployment decisions from reliance on capability demonstrations to an auditable, four-dimensional assessment of evaluation, context, compliance, and governance, validated in regulated domains to distinguish production readiness from mere functional capability.

Original authors: Fouad Bousetouane

Published 2026-07-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fouad Bousetouane

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a robot butler. You've spent months teaching it how to cook, fold laundry, and tell jokes. You run a few tests in your living room, and it looks amazing! It serves you a perfect omelet and makes you laugh. You are ready to hire it. But then you remember: this robot will be working in a busy hospital kitchen, handling real patients' food, following strict health codes, and making decisions that could affect people's lives. Just because the robot can cook doesn't mean it's ready to work there. It might not know the safety rules, it might forget to wash its hands, or it might get confused by a new menu. In the world of Artificial Intelligence, we have a similar problem. We have built "AI agents"—smart computer programs that can do tasks, use tools, and talk to us. We often judge them only on how well they perform a task, like a robot butler showing off its cooking skills. But in the real world, especially in serious places like hospitals and banks, we need to know much more than just "can it do the job?" We need to know: Is it safe? Does it follow the rules? Do we have a boss to watch it? And if it messes up, can we stop it?

This paper, titled "Stop Shipping AI Agents on Faith," argues that we are currently making a dangerous mistake. We are releasing these AI agents into the real world just because they look capable, like hiring the robot butler before checking its safety manual. The authors, Fouad Bousetouane and colleagues, say that "capability" (how smart the agent is) is not the same thing as "production readiness" (whether it is safe to actually use). To fix this, they created a new tool called the ProofAgent Index (PAI). Think of PAI as a giant, multi-part report card that doesn't just give a single grade for "cooking skills." Instead, it checks four different things:

Evaluation: Did the agent actually do the task correctly?
Context: Is the environment it's working in set up correctly? (Like making sure the kitchen has the right tools and rules).
Compliance: Did it follow all the laws and rules? (Like checking if it washed its hands).
Governance: Is there a human boss watching it, and do we have a plan if it goes wrong?

The paper introduces a system called ProofAgent Harness, which is like a testing lab that runs these agents through thousands of tricky scenarios to fill out this report card. The researchers tested their idea in two very strict worlds: healthcare (hospitals) and finance (banks). They created 12 different versions of AI agents, mixing different levels of "smartness" (from weak to strong) with different levels of "environment setup" (from messy and unprepared to clean and well-organized).

Here is what they found, and it's a big surprise for anyone who thinks "bigger and smarter is always better." They discovered that capability is not readiness. Even if you have the "strongest" AI agent in the world, if you put it in a messy, unprepared environment (what they call a "weak context"), it will fail miserably. In fact, their tests showed that a "mid-level" smart agent in a perfectly set-up environment was far more reliable than a "super-smart" agent in a messy one. The environment mattered more than the raw intelligence.

They also found that you can't just average out the scores. If an agent is great at cooking but breaks a safety rule, you can't say, "Well, it's 90% good at cooking, so it's 90% ready." The paper introduces "hard block" rules. This means if the agent fails a critical safety check or misses a required rule, the whole system automatically says "NO," no matter how good the other scores are. It's like a driver's license: you can't get a license just because you are a great driver if you don't know the traffic laws.

The researchers ran a massive test with 10,000 turns (interactions) across these different setups. They found that their new index, the PAI, was incredibly good at ranking which agents were ready and which were risky. When they used PAI to predict which configurations would fail on unseen tests, it achieved a ranking score (AUC) of 0.98. This means the index successfully ordered the agents by risk almost perfectly, distinguishing high-risk setups from low-risk ones. They also proved that simply making the AI "smarter" (using a bigger model) didn't fix the problems if the environment wasn't fixed. The biggest improvement came from "context engineering"—basically, carefully setting up the rules, instructions, and tools the AI uses.

In short, the paper suggests that we need to stop trusting AI agents just because they seem cool and capable in a demo. We need to stop shipping them on "faith." Instead, we need to use a system like the ProofAgent Index to check that they are safe, legal, and supervised before we let them loose in the real world. It's a call to move from "Look how smart this robot is!" to "Here is the proof that this robot is ready to work."

Technical Summary: Stop Shipping AI Agents on Faith: Capability Is Not Production Readiness

Problem Statement

AI agents are transitioning from controlled demonstrations to production workflows where they retrieve information, call tools, maintain state, and act on behalf of users. However, current release decisions often rely on "capability signals"—such as behavioral demos or task-specific benchmarks—that fail to capture whether an agent is ready for production constraints.

The paper argues that capability is not production readiness. A capable agent may still be unsafe to deploy if its operating context is weak, its tool use is poorly controlled, its compliance evidence is incomplete, or its governance controls are missing. In regulated domains like healthcare and finance, these gaps represent compliance, safety, and governance failures rather than mere performance defects. Existing evaluation methods measure how an agent behaves under test but do not determine if the agent has sufficient evidence to be deployed, monitored, and governed within a defined operating scope.

Methodology: The ProofAgent Index (PAI)

To address the gap between behavioral performance and deployment readiness, the authors introduce the ProofAgent Index (PAI), a governance readiness index. PAI reframes agent release as an auditable decision based on decomposable evidence rather than faith.

1. Four Readiness Dimensions

PAI aggregates evidence across four normalized dimensions (scaled 0–100):

Evaluation (E): Measures observed agent behavior under adversarial and expected conditions (e.g., task success, hallucination resistance, safety, tool use).
Context (Q): Measures the quality of the operating environment, including role clarity, instruction consistency, grounding, tool schemas, guardrails, and injection hardening.
Compliance (C): Measures whether observed behavior and artifacts satisfy applicable legal, regulatory, and policy obligations (e.g., GDPR, HIPAA, EU AI Act). Missing evidence is treated as unsatisfied.
Governance (G): Measures organizational controls, including ownership, scope definition, approval workflows, monitoring, incident response, and rollback mechanisms.

2. Aggregation and Admissibility Gate

PAI is computed in two stages to prevent "averaging away" critical defects:

Geometric Mean Aggregation: The raw score ( $PAI_{raw}$ ) is a weighted geometric mean of the four dimensions. This ensures that a low score in one dimension pulls the total down more sharply than an arithmetic mean, limiting compensation between dimensions.
Admissibility Gate (Hard Blocks): A "cap" is applied based on hard block rules. If a configuration triggers a prohibited use condition, critical safety failure, missing mandatory compliance evidence, or unresolved governance finding, the final score is capped below the review threshold regardless of the raw geometric mean. This ensures non-negotiable defects block release.

3. Readiness Bands

The final score is mapped to operational bands:

Blocked: Hard block condition triggered; release prohibited.
Not Ready: Does not meet minimum readiness bar.
Ready with Caveats: Meets review threshold but requires monitoring or remediation.
Ready: Satisfies full threshold with no blocking conditions.

Empirical Validation

The authors validated PAI using ProofAgent Harness, an open-source infrastructure for auditable evaluation. The study covered two heavily regulated domains (healthcare and finance) across a configuration grid of 12 units (3 capability tiers × 2 context conditions × 2 domains).

Experimental Setup

Capabilities: Weak (8B model), Mid (70B model), and Strong (Mixture-of-Experts model).
Context: Baseline (B0, weak instructions/governance) vs. Engineered (B5, strong instructions/governance).
Validation Strategy: A "held-out" approach where PAI scores were computed from a development trap pack (25 turns), while the PASS/FAIL outcome was measured on a disjoint exam pack (10 turns) to prevent circular validation.
Scale: A 10,000-turn validation across 1,000 sessions to analyze defect patterns.

Key Results

Predictive Power: PAI achieved an AUC of 0.98 in ordering configurations by their risk of failure on unseen traps. This indicates strong discrimination between ready and non-ready configurations.
Context vs. Capability:
- Context Engineering: Had the strongest effect. Moving from weak (B0) to engineered (B5) context reduced defect rates from 65.74% to 17.84% (a 72.9% relative reduction).
- Capability Saturation: While moving from weak to mid capability significantly reduced defects, the jump from mid to strong yielded diminishing returns. Crucially, a mid-capability agent in a strong context (B5) was far more reliable than a strong-capability agent in a weak context (B0).
Capability is Not Readiness: The study confirmed that high behavioral capability does not guarantee readiness. Weak agents remained risky even under strong context conditions, while strong agents failed under weak governance or context.
Failure Modes: Defects spanned safety, tool use, hallucination resistance, and phantom tool calls, reinforcing the need for a readiness framework broader than simple task accuracy.

Key Contributions

Definition of Readiness: The paper defines AI agent production readiness as a governance problem extending beyond behavioral evaluation, requiring evidence of safe operation within a specific risk tier and compliance scope.
The PAI Framework: Introduction of a four-dimensional index (Evaluation, Context, Compliance, Governance) that integrates technical behavior with organizational controls.
Implementation: Description of PAI's implementation in ProofAgent Harness, providing an open-source CLI for generating decomposable readiness artifacts, hard block decisions, and release reports.
Empirical Evidence: Validation showing that PAI carries a held-out readiness signal and that context engineering and governance evidence are critical determinants of deployment risk, often outweighing raw model capability.

Significance and Claims

The paper claims that PAI reframes agent release from a "faith-based" decision into an auditable readiness decision. Its significance lies in providing a unified language for three distinct stakeholders:

Engineers: Receive measurable, decomposable signals (not just a single behavioral score).
Risk/Compliance Teams: Receive auditable evidence linking behavior to policy and control obligations.
Business Leaders: Receive interpretable release decisions that account for context, compliance, and governance, rather than just model performance.

The authors conclude that AI agents should not be shipped because they appear capable; they should only be shipped when there is evidence that they are ready to operate inside a governed production environment. PAI serves as the mechanism to convert agent evaluation from an act of confidence into a verifiable readiness decision.