💬 NLP

The Checking Problem: What must be true before AI ships in a regulated firm

This paper demonstrates that the viability of enterprise AI in regulated financial services depends less on raw accuracy and more on measurable attributes like verifiable attribution and confidence signaling, which can significantly reduce the human review burden required for production deployment.

Original authors: Prerit Ahuja

Published 2026-08-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Prerit Ahuja

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a spaceship, and you've just hired a brand-new, super-smart robot co-pilot to help you navigate. You've run a quick test: you asked the robot to point to a star on a map, and it did it perfectly. You high-five, the crew cheers, and you're ready to launch. But here's the catch: in the real world of space travel (or in the high-stakes world of banking and finance), a single perfect test isn't enough. You need to know if the robot can point to that same star every single time you ask, if it can show you exactly where it saw the star in the map, and if it can honestly tell you, "I'm not sure about this one," when it's confused.

This is the heart of a new study by researcher Prerit Ahuja, which dives into the messy reality of "Enterprise AI." Think of this as the science of figuring out why so many companies try to use AI to do their paperwork and then hit a wall. The paper focuses on a few key ideas: Accuracy (getting the right answer), Reproducibility (getting the same answer twice), Groundedness (being able to show your work), and Detectability (knowing when you're wrong). The big question isn't just "Is the AI smart?" but "Is the AI safe enough to let it work without a human watching its every move?"

The "Demo Trap" vs. The "Real Deal"

The paper starts by pointing out a sneaky problem called the "Demonstration Trap." Imagine you show a magic trick to a group of investors. You pull a rabbit out of a hat, and it's perfect. They are impressed. But if you try to do that same trick 100 times in a row, maybe the rabbit gets tired, or the hat gets stuck, or you accidentally pull out a shoe instead.

In the world of regulated finance (where rules are strict and mistakes cost millions), companies often let AI pass a "Demonstration Bar." This bar is low: the AI just has to get one answer right on one document, once. The study found that 57 out of 72 different AI setups passed this easy test. It looked like a win!

But then, the paper raises the "Production Bar." This is the real test. To clear this bar, an AI must:

Be right 99% of the time, every single time.
Give the exact same answer if you ask it the same question twice.
Cite its sources perfectly (showing exactly where it found the info).
Give a confidence signal that actually means something (telling a human, "I'm 90% sure this is right" or "I'm guessing").

When the researchers tested the same AI setups against this harder bar, the results were a shock. Only 32 out of 72 configurations made the cut. That means 43.9% of the AI tools that looked perfect in the demo failed completely when asked to do the real job. The "magic trick" didn't work when the lights were on and the audience was watching closely.

The Cost of Checking: Why "Right" Isn't Enough

Here is the most surprising part of the paper. Even if an AI is mostly right, it might still be useless if a human has to check everything it does.

The researchers measured the "Review Burden." Imagine a human reviewer is a gatekeeper. If the AI says, "I'm 99% sure this is correct," the gatekeeper might skip checking it. But if the AI says nothing about how sure it is, the gatekeeper has to check 100% of the work.

The study tested three different ways to build these tools:

The Plain Tool (C1): Just does the task. Result? It offers no confidence signal. The human has to check 100% of the output. It's like hiring a robot that never speaks, so you have to read every word it types.
The Governed Tool (C2): This tool is forced to show its work and say how confident it is. Result? The human only needs to check 49% of the output. That's a huge win! The tool isn't necessarily more accurate, but it tells the human which parts are safe to ignore.
The Verified Tool (C3): This tool does the task, then reads its own work again to fix mistakes. You might think this is the best. But the study found it was actually the worst deal. It took 2.3 times longer to run, and it only reduced the check rate to 44%. Worse, it was the only setup that failed to keep the error rate low enough.

The "Reconciliation" Surprise

One of the most vivid examples in the paper involves a task called "Reconciliation." Imagine you have two lists of numbers (like bank balances) and you need to see if they match.

One AI model (Open-weights A) looked amazing in the demo. It could spot if there was a mismatch.
But when the researchers asked it to say how much the numbers were off, it was basically guessing. It got the right answer only 31.9% of the time on this specific task, even though it looked great on other tasks.

This proves that a tool can be a "one-trick pony." It might pass a demo by spotting a problem, but fail the production test because it can't calculate the solution.

What This Means for the Future

The paper concludes with a simple but powerful shift in how we should think about AI.

Don't just ask: "How often is it right?"
Ask instead: "Does it know when it's wrong? Can it show its homework? How much of its work will a human still have to check?"

The study suggests that the value of an AI isn't just about how smart it is, but how much it saves humans from doing boring, repetitive checking. If an AI can't tell you when it's unsure, you have to check everything, and that costs too much time and money.

The researchers are careful to say this is a "first measurement" and that real-world documents are messier than the ones they tested, so the problems might be even bigger in reality. But the message is clear: before we let AI ship into the real world, we need to stop looking at the "magic trick" and start checking the math, the sources, and the confidence. Because in the end, a tool that is right 99% of the time but makes you check 100% of its work is just a very expensive, very slow typewriter.

Technical Summary: The Checking Problem

1. The Problem

Enterprise AI programmes in regulated sectors, particularly financial services, exhibit high attrition rates. While common explanations cite model capability, data quality, or integration difficulties, this paper proposes a different mechanism: the discrepancy between the criteria used to approve a pilot (the "demonstration bar") and the criteria required for regulated deployment (the "production bar").

A pilot is often judged on a single correct run on a single document. This exercise fails to test critical properties required for production: reproducibility across repeated runs, the verifiability of cited sources (groundedness), and the ability of the model to signal when it is wrong (detectability). Consequently, tools that pass steering committee demonstrations often fail to ship because these properties are structurally invisible at the pilot stage but decisive in production.

2. Methodology

The study measured this gap using a controlled experiment across six document-heavy workflows typical of regulated financial services.

Workflows and Corpus

Six workflows were defined, ranging from low-judgement extraction to high-judgement reconciliation:

W1–W4 (Synthetic): Fixed-field extraction, entity screening, cross-document reconciliation, and regulatory categorisation. These used synthetic documents generated deterministically from a fixed seed using standard Python libraries. This approach ensured exact ground truth, eliminating the risk of human error in establishing the "correct" answer key.
W5–W6 (Real): Attributed summarisation and multi-document comparison. These used genuine filings retrieved from SEC EDGAR.

Tool Configurations

Three configurations representing increasing levels of engineering effort were tested:

C1 (Plain): The model performs the task.
C2 (Governed): The model cites source passages, states a confidence score for each element, and declines to answer if unsupported.
C3 (Verified): C2 followed by a self-verification pass where the model re-reads its output against the source to issue corrections.

Models and Scoring

Four model families (frontier closed, mid-tier closed, and two open-weights) were evaluated. Each configuration was run three times at temperature zero. Scoring was deterministic for W1–W4 and mechanical for W5–W6 (checking if cited passages exist in the source), removing judge variance.

Acceptance Bars

Demonstration Bar: One run, one case, all elements correct.
Production Bar: Four criteria must hold:
1. Accuracy: Element-level correctness $\ge$ 0.99 across every case and repeat.
2. Reproducibility: Identical input yields identical output across repeats.
3. Groundedness: Cited passages exist in the source document (threshold 0.95).
4. Detectability: Stated confidence separates correct from incorrect elements (AUC $\ge$ 0.70).

Review Burden Estimation

The paper calculates the "review burden"—the share of output a human must inspect to maintain a 1% residual error tolerance. This was estimated out-of-sample (calibrating on a pilot sample, applying to held-out data) rather than using hindsight (oracle) figures.

3. Key Results

The Survival Gap

57 of 72 configurations (79%) cleared the demonstration bar.
32 of 72 configurations (44%) cleared the production bar.
Survival Rate: 56.1%. This means 43.9% of configurations that pass a demonstration cannot be placed in production.

Failure Analysis

Among the 25 configurations that passed the demonstration bar but failed the production bar:

Accuracy was the dominant failure mode (22 of 25), but it was not the sole cause.
Reproducibility was the sole cause of failure in 3 configurations, more often than accuracy alone.
Detectability failed in 15 configurations, and Groundedness in 6.
Workflow Specificity: Fixed-field extraction (W1) achieved 1.000 accuracy across all models. Reconciliation (W3) was the weakest at ~0.80. The ability to automate is workflow-dependent, specifically distinguishing between tasks requiring arithmetic over documents versus rule application.

The Impact of Governance (C2) and Verification (C3)

Accuracy: Moving from C1 to C3 did not significantly improve accuracy (mean change -0.0168, p=0.118) or reproducibility.
Groundedness: Self-verification (C3) improved groundedness from 0.933 to 0.952.
Latency: C3 increased latency by 2.3x compared to C1.
Review Burden:
- C1 (Plain): Requires 100% review. Without a confidence signal, a reviewer cannot triage.
- C2 (Governed): Reduces review burden to 49% while holding error tolerance in 17 of 20 configurations.
- C3 (Verified): Reduces burden to 44% but is the only configuration that fails to hold the error tolerance in the out-of-sample test.

The Oracle vs. Operational Gap

The "oracle" figure (knowing errors in hindsight) understates the actual review burden by roughly 2.7 times. A tool that emits no confidence signal forces a 100% review rate, regardless of its underlying accuracy.

4. Significance and Claims

The paper does not claim to have solved AI deployment or to have established a universal standard. Its contributions are threefold:

An Instrument: A pair of acceptance bars (demonstration vs. production) published in full to allow dispute and application.
A Measurement: Empirical evidence that a significant portion of "successful" pilots fail production readiness due to invisible properties (reproducibility, groundedness, detectability).
A Costing: A quantification of the "review burden."

Core Implication: The value of an AI workflow in a regulated firm is set less by how often the model is right, and more by how much of its output a human must still check. This second property is measurable but rarely measured.

Practical Shifts:

Vendor Evaluation: Questions should shift from "How accurate is it?" to "Does it know when it is wrong, and can its citations be traced?"
Pilot Criteria: Pilots should include repeated runs and source verification checks, which cost little but test critical production properties.
Economic Case: The decision to deploy is driven by the review ratio. If checking is cheaper than producing, a tool pays even with full review. If checking costs nearly as much as deciding (common in judgement-heavy work), the tool may not be viable.

The paper acknowledges limitations: the synthetic corpus is cleaner than real data (making the measured gap a lower bound), the configurations are a proxy rather than a census of all tools, and the sample size is modest. It frames the work as a "first measurement" of a previously unmeasured quantity.