💻 computer science

Healthcare LLM Benchmarks Are Only as Good as Their Explicit Assumptions

This paper argues that the gap between healthcare LLM benchmark performance and real-world deployment stems from untested implicit assumptions about user behavior, proposing a framework to classify these assumptions and introducing "BenchmarkCards" and staged evaluation to systematically address them.

Original authors: Naveen Raman, Santiago Cortes-Gomez, Mateo Dulce Rubio, Fei Fang, Bryan Wilder

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Naveen Raman, Santiago Cortes-Gomez, Mateo Dulce Rubio, Fei Fang, Bryan Wilder

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "Driving Test" vs. "Real Traffic"

Imagine you are learning to drive. You take a driving test on a quiet, empty track with a perfect instructor giving you clear, written instructions. You pass with flying colors. The test says you are a "perfect driver."

But then, you get behind the wheel in a real city. Suddenly, people are jaywalking, the weather is bad, your passengers are yelling directions, and you have to make split-second decisions. You crash.

The paper argues that Healthcare AI (LLMs) are currently in this exact situation. We have "driving tests" called benchmarks where AI models score 95% or higher. But when we put them in real hospitals with real patients, their performance drops drastically.

The authors say: The problem isn't that the AI is bad or the test is poorly written. The problem is that the test is based on hidden guesses (assumptions) that don't hold up in the real world.

The Two Types of Hidden Guesses

The authors break these hidden guesses into two categories. Think of them as the "Rules of the Game" and the "Rules of the Outcome."

1. Task Assumptions (The "Rules of the Game")

These are guesses about how the AI interacts with people during the test.

The Assumption: The test assumes patients will type perfect, complete sentences, just like a doctor writing a report.
The Reality: Real patients are messy. They might forget symptoms, get confused, ask follow-up questions, or type in broken sentences.
The Fix: This is easy to test. We just need to look at real conversations and see if the test matches them. If the test is too "clean," we can redesign the test to be messier and more realistic.

2. Outcome Assumptions (The "Rules of the Outcome")

These are guesses about what happens after the AI gives an answer.

The Assumption: The test assumes that if the AI gives the "right" medical answer, the patient will actually follow it and get better.
The Reality: Humans are unpredictable. A patient might ignore the AI's advice, misunderstand it, or decide to do something else entirely. The AI might be "right," but if the human doesn't listen, the health outcome is still bad.
The Fix: You cannot fix this with a better computer test. You cannot simulate human behavior perfectly on a screen. To test this, you have to do real-world experiments (like clinical trials) where you watch what actual people do.

The "Halfway" Discovery

The authors looked at a real medical study to see how much of the performance drop was caused by which type of guess.

The Result: They found the gap between the "perfect test score" and the "real-world failure" was split right down the middle.
- 50% was because the test didn't match how people talk (Task Assumptions).
- 50% was because the test didn't account for how people actually behave (Outcome Assumptions).

This is a big deal because it means no matter how perfect we make the computer benchmarks, we can only ever solve half the problem. The other half requires real-world human testing.

The Proposed Solutions

To fix this, the authors suggest two new tools:

1. BenchmarkCards (The "Ingredient Label")

Right now, when a new AI test is released, it's like buying a cake mix without a label. You don't know what's in it or who it's for.
The authors propose BenchmarkCards. These are simple documents attached to every test that explicitly list the "hidden guesses."

Example: "This test assumes the user is a doctor, not a patient." or "This test assumes the user will only ask one question."
Why it helps: Before a hospital uses a test, they can look at the card and say, "Oh, this test assumes patients are perfect. That doesn't fit our hospital. We can't trust these results."

2. Staged Evaluation (The "Training Camp")

Instead of jumping straight to a full hospital rollout, the authors suggest a step-by-step process:

Start with the Benchmark: Get the initial score.
Check the "Task" Guesses: Test the AI with real, messy patient conversations. If it fails, fix the model or the test.
Check the "Outcome" Guesses: If the AI passes the conversation test, run small, real-world studies to see if people actually listen to the advice and get better.
Deploy: Only release the AI when you have proven that both the conversation style and the human behavior match your expectations.

Summary

The paper argues that we are trying to predict how AI will work in the real world by looking at a static picture. But healthcare is a movie, not a photo.

To bridge the gap, we need to stop pretending our tests are perfect. We need to write down our assumptions (BenchmarkCards) and test them in stages, acknowledging that some things (like human behavior) can only be proven by watching real people, not by running code.

Technical Summary: Healthcare LLM Benchmarks Are Only as Good as Their Explicit Assumptions

Problem Statement

The paper addresses the persistent "evaluation–deployment gap" in healthcare Large Language Models (LLMs). While high performance on benchmarks is often cited as evidence of clinical readiness, recent studies demonstrate a significant discrepancy between benchmark results and real-world deployment performance. The authors argue that this gap does not stem from poorly designed benchmarks or insufficient model capabilities, but rather from implicit assumptions embedded in evaluation protocols. These assumptions concern how users interact with models and how model outputs translate into real-world decisions. Because these assumptions are left unstated, practitioners cannot determine if benchmark results are transferable to their specific deployment contexts, leading to situations where strong benchmark performance coexists with weak or even harmful deployment outcomes.

Methodology

The authors propose a framework to operationalize the identification and testing of these implicit assumptions. The methodology consists of three core components:

1. Classification of Assumptions

The paper categorizes the assumptions underlying the evaluation–deployment gap into two distinct types based on the data required to test them:

Task Assumptions: Concern the relationship between the conversation data in the evaluation environment versus the deployment environment. These include assumptions about query distribution (e.g., expert-written vs. patient-written), interaction type (single-turn vs. multi-turn), and information completeness. These can be tested using conversation data alone.
Outcome Assumptions: Concern whether the benchmark's evaluation criterion (often a proxy outcome) tracks the actual decision-making target in the real world. These involve data beyond the conversation itself, such as patient behavior, adherence to recommendations, or true clinical outcomes. These cannot be tested or resolved through benchmarks alone and require behavioral studies or Randomized Controlled Trials (RCTs).

2. Sensitivity Analysis and Gap Decomposition

To demonstrate the framework's utility, the authors perform a retrospective sensitivity analysis on a healthcare RCT (Bean et al., 2026). They decompose the total performance gap ( $\Gamma$ ) by sequentially relaxing assumptions:

They start with the benchmark performance (95% accuracy).
They relax task assumptions (e.g., switching from expert-written to patient-written queries, and from single-turn to multi-turn interactions), observing a performance drop to 64%. This attributes approximately 31 percentage points of the gap to task assumptions.
They attribute the remaining drop (from 64% to the final deployment performance of 34%) to outcome assumptions, accounting for the other ~30 percentage points.
This analysis illustrates that even if all task assumptions are addressed, a significant gap remains due to outcome assumptions that require real-world validation.

3. Proposed Artifacts and Protocols

To address these gaps, the authors propose two concrete mechanisms:

BenchmarkCards: A structured documentation artifact (analogous to Model Cards) that benchmark designers must complete. These cards explicitly list the task and outcome assumptions encoded in the evaluation protocol. Practitioners use these cards to assess whether a specific benchmark's assumptions hold in their deployment context.
Staged Evaluation: A systematic evaluation procedure that iterates through the following steps:
1. Compare BenchmarkCards against the intended deployment context to identify gaps.
2. Collect data to test task assumptions (e.g., gathering real user interaction logs).
3. Test task assumptions and refine models or benchmarks until task-related gaps are minimized.
4. Rank and test outcome assumptions using behavioral studies or RCTs, as these cannot be resolved by benchmark tuning alone.
5. Deploy only when the performance drop due to unsatisfied assumptions is sufficiently low.

Key Contributions

Conceptual Framework: The paper introduces a dichotomy between task and outcome assumptions, clarifying that while benchmarks can address task assumptions, outcome assumptions inherently require real-world behavioral data and cannot be solved by better benchmark design alone.
BenchmarkCards: A proposal for a standardized documentation format that forces the explicit declaration of assumptions, enabling practitioners to identify mismatches between evaluation and deployment before model selection.
Staged Evaluation Protocol: A procedural guide for healthcare LLM evaluation that moves beyond static benchmarking to a dynamic process of assumption testing, integrating conversation data analysis with behavioral studies and RCTs.
Empirical Demonstration: A retrospective analysis of a real-world RCT showing that the evaluation–deployment gap is roughly split evenly between task and outcome assumptions, proving that benchmark improvements alone can only close about half of the gap.

Results

The retrospective analysis of the Bean et al. [2026] study serves as the primary empirical result. The authors found that:

The total performance gap between the benchmark (95%) and deployment (34%) was 61 percentage points.
Relaxing task assumptions (query distribution and interaction type) accounted for a 31 percentage point drop.
Relaxing outcome assumptions (specifically the link between LLM recommendations and actual patient decisions) accounted for the remaining 30 percentage point drop.
This result empirically validates the claim that well-designed benchmarks are insufficient for predicting deployment performance because they cannot capture outcome assumptions dependent on human behavior.

Significance and Claims

The paper claims that the core issue in healthcare LLM evaluation is not the quality of benchmarks, but the validity of the assumptions they rely on. By making these assumptions explicit, the field can move from a paradigm where "better benchmarks" are assumed to solve all problems to one where evaluation is a structured process of validating transferability.

The authors argue that:

Benchmarks are necessary but insufficient: They provide a starting point but cannot predict deployment performance without explicit assumption testing.
Outcome assumptions are the bottleneck: Since these rely on human behavior and real-world outcomes, they necessitate RCTs and behavioral studies, similar to clinical drug trials.
Transparency is the solution: Tools like BenchmarkCards and staged evaluation protocols provide a pathway to safe deployment by ensuring practitioners understand exactly where and why a benchmark might fail in their specific context.

The paper concludes that closing the evaluation–deployment gap requires a shift from purely model-centric evaluation to an assumption-centric approach that integrates conversation data analysis with real-world outcome validation.