📄 medicine

Information limits constrain medical AI claims beyond model sophistication

The paper introduces the AI Evaluation Protocol (AEP) to demonstrate that medical AI claims are fundamentally constrained by the information content and structural branching of the deployment data itself, rather than solely by model sophistication, often revealing that apparent performance stems from substrate structure rather than genuine predictive signal.

Original authors: M. Antony Ewing

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: M. Antony Ewing

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The Map vs. The Terrain

Imagine you are trying to navigate a city using a GPS app. Usually, we judge the GPS by how well it gets us to our destination on a sunny day. But what if the GPS works perfectly on a clear map, yet fails completely when the roads are blocked by construction or fog?

This paper argues that in medical AI, we are currently obsessed with how "smart" the GPS (the AI model) is, but we are ignoring the terrain (the patient data) it is trying to navigate.

The author, Maurice Antony Ewing, introduces a new way to test medical AI called the AI Evaluation Protocol (AEP). Instead of asking, "Is this model smart?", AEP asks, "Does the data we have actually contain enough information to solve this problem?"

The Core Problem: "Predictive Blindness"

The paper explains that sometimes, two patients look exactly the same on paper (same age, same blood pressure, same symptoms), but they have completely different futures. One might get better, and the other might get worse.

The Analogy: Imagine a weather forecaster looking at a sky that is 50% sunny and 50% stormy. No matter how powerful the forecaster's computer is, they cannot predict the weather for this specific moment because the sky itself is ambiguous.
The AI's Dilemma: When an AI sees two patients who look identical but have different outcomes, it gets "confused." To make a guess, it simply picks the most common outcome (the "majority branch"). If 60% of similar patients get better, the AI guesses "better" for everyone.
The Trap: The AI might look very accurate overall because it's just following the crowd. But for the 40% of patients who are different, the AI is essentially guessing blindly. The paper calls this "Predictive Blindness Risk."

The New Test: The "Floor" Check

The paper proposes a new way to test AI claims. Instead of just looking at the final score (like a test grade), the AEP checks if the AI is beating the "Local Floor."

The Floor: This is the accuracy you get if you just guess the most common outcome for a specific group of patients. It's the "lazy" baseline.
The Test: The AEP splits the data into three zones:
1. Clear Zones: Where the data clearly predicts the outcome (easy).
2. Mixed Zones: Where the data is ambiguous.
3. Blind Zones: Where the data is so confusing that even the "lazy guess" is a coin flip.

The paper's main finding is that many AI models look great in the "Clear Zones" but fail to do anything better than the "lazy guess" in the "Blind Zones."

The Results: Smart Models, Empty Data

The author tested this protocol on four different types of medical data (like memory tests for Alzheimer's, ICU vital signs, tumor images, and population health surveys) using 12 different types of AI models.

Here is what they found:

The Illusion of Skill: Some models had high scores (AUC of 0.90), which usually means they are excellent. However, when the AEP looked at the "Blind Zones," these models had zero advantage over just guessing the majority outcome. They were riding on the structure of the data, not their own intelligence.
The "False Appearance": In one specific case (acute care vital signs), the model looked very accurate, but the paper found a high "False Appearance Index." This means the model looked smart only because the data was easy in some places, but it couldn't actually help in the hard, ambiguous situations where doctors need it most.
The Good News: It's not that AI never works. In a few specific tasks (like predicting changes in kidney function markers), the models did find extra information that allowed them to beat the "lazy guess" even in the hard zones. These claims were "supported."

The Conclusion: "Supported" vs. "Unproven"

The paper concludes that we need to stop assuming that a high score on a test means the AI will work in the real world.

If the data is ambiguous (like a foggy road), and the AI just guesses the majority, it is not providing medical value, even if its overall score is high.
The Verdict: The AEP gives a simple verdict for any medical AI claim:
- Supported: The AI found real information in the hard-to-solve cases.
- Bounded/Weak: It helped a little, but not enough to be fully trusted.
- Failed: It didn't do better than a simple guess in the hard cases.
- Untested: We don't have enough data to know if it works or not.

In short: The paper argues that medical AI is limited by the quality of the data, not just the complexity of the code. If the patient data doesn't contain the answer, no amount of "sophisticated" AI can invent it. We need to check if the data supports the claim before we trust the model.

Technical Summary: Information Limits Constrain Medical AI Claims Beyond Model Sophistication

Problem Statement

Medical AI claims are typically grounded in the reported performance of large, often proprietary models on clinical tasks. However, external observers cannot inspect the inner workings of these models (weights, training substrates, calibration logic, or data-generating restrictions). Existing evaluation frameworks (e.g., TRIPOD+AI, CONSORT-AI, PROBAST+AI) focus on how well a study is conducted or reported, presupposing access to the model. They do not address a fundamental gap: whether the deployment data itself contains sufficient information to support a specific claim, regardless of model sophistication.

The core problem is that clinical AI models perform a many-to-one compression of patient data ( $x$ ) to outcomes ( $y$ ). If a specific data configuration $x_1$ maps to multiple opposing future states ( $y_1$ and $y_2$ ) due to unobserved latent factors (e.g., treatment effects, unrecorded biology), the outcome is non-identifiable from $x$ . In such regions, a model may appear accurate by simply siding with the dominant branch (the local majority), creating a "false appearance" of skill without resolving the underlying uncertainty. This leads to a decoupling of benchmark performance (AUC) from prospective reliability in deployment.

Methodology: The AI Evaluation Protocol (AEP)

The authors introduce the AI Evaluation Protocol (AEP), a measurement framework that evaluates the substrate (the data environment) rather than the model. AEP operates independently of model architecture and can be applied even when model internals are inaccessible.

Core Concepts

Identifiability Partitioning: AEP partitions a prospective dataset into three regions based on non-parametric, pre-model diagnostics:
- Identifiable: Data states where the outcome is locally determined (one-sided future).
- Mixed: Data states with some branching.
- Non-Identifiable: Data states where matched states branch into opposing futures.
Diagnostics:
- Predictive Blindness Risk (PBR): Measures the local mass of opposing future branches within matched-state neighborhoods. $PBR = 1$ indicates a maximally balanced split between opposing outcomes; $PBR = 0$ indicates a one-sided outcome.
- Predictive Indeterminacy Measure (PIM): Measures the normalized separation between opposing branches in outcome units.
Performance Metrics:
- Local Majority Floor: The accuracy achievable by simply predicting the dominant branch in a non-identifiable region.
- Residual Predictive Advantage (RPA): The accuracy a model retains above the local majority floor in non-identifiable regions.
- Claim Support Score (CSS): A composite score combining global performance and restricted-space support. If RPA is zero or negative, the CSS is zero, regardless of global AUC.

Theoretical Basis

The protocol relies on a Local Substrate Floor Bound Theorem. It posits that if a model cannot distinguish between branch-separating latent variables within a matched-state cell, its maximum directional accuracy is bounded by the local majority probability ($1 - PBR/2$). Any claim of predictive skill in these regions requires evidence that the model exceeds this floor.

Experimental Design

The authors applied AEP across four distinct clinical substrates representing different information geometries:

ADNI: Routine longitudinal monitoring (cognitive direction).
MIMIC-IV: Acute-care physiology (direction prediction for lactate, MAP, creatinine, etc.).
Lumiere/Yale: Exploratory tumor imaging.
HRS: Population-health follow-up.

A "12-model architecture-agnostic zoo" (including logistic regression, random forests, gradient boosting, etc.) was run on ADNI and MIMIC-IV to ensure verdicts reflected substrate structure rather than specific algorithmic biases. The study comprised ~40,500 unique patients and over 2 million state transitions.

Key Results

The study analyzed 96 model-claim evaluations across the datasets. The findings demonstrate a clear separation between apparent global performance and information-constrained support.

Decoupling of Performance and Reliability:
- Apparent discrimination (Median AUC) ranged from 0.66 to 0.90 across substrates.
- However, Residual Predictive Advantage (RPA) was at or below zero in every substrate except one (MIMIC-IV acute-state trajectory).
- In the Lumiere/Yale imaging substrate, which showed the highest apparent discrimination (AUC 0.900), the RPA was negative (-0.012), indicating the model's performance was driven by substrate structure rather than resolving the specific clinical transition.
Substrate-Specific Outcomes:
- Supported Claims: Only specific tasks in MIMIC-IV (urea nitrogen, creatinine, MAP, and acute-state trajectory) showed models retaining positive residual advantage over the local floor in non-identifiable regions.
- Bounded/Failed Claims: Tasks like lactate, bilirubin, and white blood cells showed global signals that did not translate to restricted-space support. For example, in the bilirubin task, the best model had a global AUC of 0.682 but failed to beat the local floor in non-identifiable regions (RPA = 0.031, categorized as "bounded").
- Untested Claims: The ADNI CDR-SB task had high global AUC but lacked a severe non-identifiable test region under the prespecified threshold, resulting in an "untested" verdict rather than a false sense of security.
False Appearance Index:
- The MIMIC-IV acute-state trajectory had the highest "false-appearance index" (0.627), meaning it exhibited high apparent performance while carrying significant risk of being driven by dominant branch momentum rather than resolved information.
Stability:
- Repeated splits of the MIMIC-IV data confirmed that global discrimination was stable, but the divergence between global AUC and restricted-space support was a consistent property of the substrate, not an estimation artifact.

Significance and Claims

The paper does not claim that medical AI is universally ineffective. Instead, it establishes that medical AI capability is bounded by the information contained in the deployment substrate, not just by model sophistication.

Protocol Utility: AEP provides a method to evaluate the validity of AI claims when model internals are inaccessible. It shifts the burden of proof: if a substrate is locally non-identifiable, the claimant must provide evidence of residual advantage in that region.
Distinction of Verdicts: The protocol distinguishes between claims that are supported (exceed local floor), bounded (weak support), failed (no support despite global metrics), and untested (insufficient data to test the claim).
Limitation of Global Metrics: The authors argue that global metrics like AUC are insufficient for establishing clinical value in ambiguous regions. A model can be "reliably poor" (stable but low accuracy) or "unreliably good" (high global AUC but zero residual advantage in critical non-identifiable regions).
Clinical Implication: The findings suggest that in many clinical scenarios, the "ceiling" of a model is the recorded substrate, whereas a clinician's intuition may access richer, unrecorded context. Therefore, trust in medical AI must be conditional on the substrate's ability to support the specific claim, not just the model's benchmark scores.

The paper concludes that AEP serves as a necessary prior evidentiary layer, ensuring that claims are not over-interpreted based on global performance when the underlying data structure renders the specific prediction indeterminate.