💻 computer science

Retrieval, not hallucinations, will be the limiting factor for LLM-based clinical AI tools

This paper argues that for LLM-based clinical AI tools, the primary limitation is not hallucinations but rather the failure to accurately retrieve necessary patient-level data, urging a shift in focus toward improving recall and retrieval evaluation.

Original authors: Kirk Roberts, Steven Bedrick, Kurt Miller, William R. Hersh, Hongfang Liu

Published 2026-07-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Kirk Roberts, Steven Bedrick, Kurt Miller, William R. Hersh, Hongfang Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery, but instead of a detective, you have a super-smart robot assistant. This robot has read almost every book ever written and can talk like a human. In the world of medicine, these robots are called Large Language Models (LLMs). They are being taught to help doctors by reading patient records, summarizing what's wrong, and suggesting treatments. But here's the catch: the robot can't read the entire hospital database at once because it's too huge. So, before it answers a question, it has to send out a "search party" to find the specific pages of the patient's history that matter. This process is called Retrieval.

Think of the robot as a brilliant chef and the patient's medical history as a massive, chaotic pantry. The chef (the AI) is amazing at cooking, but if the search party (the retrieval system) forgets to bring back the eggs, the chef can't make an omelet, no matter how talented they are. For a long time, everyone was worried that the chef might just make up ingredients that don't exist—like claiming they used "unicorn eggs" when there were none. This is called a hallucination. But this paper suggests that while making things up is bad, the real, silent danger is the chef missing the real ingredients because the search party didn't find them. If the robot forgets to tell the doctor that a patient is allergic to penicillin because it couldn't find that note in the giant pantry, the consequences could be much worse than a silly made-up fact.

The Paper's Big Idea: The Silent Killer in the Pantry

This paper argues that the biggest bottleneck for medical AI isn't the robot making things up (hallucinations); it's the robot failing to find the right information in the first place (recall errors). The authors, a team of experts from universities and clinics, want to shift the conversation from "Is the AI lying?" to "Did the AI miss something critical?"

The Two Types of Mistakes

To understand their point, the authors compare AI errors to two classic types of mistakes:

The "False Alarm" (Precision Error): This is when the AI says something is true, but it's actually wrong. In the medical world, this is the famous hallucination. For example, the AI might say, "The patient has a rare tropical disease," when they actually don't. This is scary, but it's usually easy to spot. If the AI says something wild, a doctor can look at the source note and say, "Wait, that's not in the record." It's like the chef claiming to use a secret spice that isn't in the pantry; you can just check the pantry and see it's missing.
The "Silent Omission" (Recall Error): This is when the AI leaves out something important that is actually in the record. Maybe the patient has a severe allergy, or a critical lab result from yesterday, but the AI just doesn't mention it. This is the "silent killer." Why? because if the AI doesn't say anything, the doctor might assume the information doesn't exist. It's like the chef forgetting to mention the eggs are missing, so the doctor assumes the omelet is fine, only to find out later the patient had a reaction to a hidden ingredient.

The paper points out that while we have good ways to catch the "False Alarms" (by checking the source notes), we have almost no way to catch the "Silent Omissions." If the search party fails to find a specific note, the AI never sees it, and the doctor never knows it was missed. It's a silent failure.

Why the Search Party is the Weak Link

The authors explain that while the "chef" (the LLM) is getting smarter and faster every day, the "search party" (the retrieval system) isn't improving nearly as fast.

The Chef's Growth: The AI models are evolving rapidly, getting better at understanding language and context.
The Search Party's Stagnation: The tools used to search through millions of medical notes are still relying on older, slower methods. Even though we use fancy new tech to help them, the core way they find documents hasn't changed much.

The paper suggests that because the search tools are lagging behind, they are becoming the weak link. As patient records get longer and more complex (thanks to better technology connecting different hospitals), the search party has an even harder job. If they miss a single crucial note in a massive file, the whole AI system fails, and no one realizes it until it's too late.

The Evaluation Trap

One of the most interesting parts of the paper is how hard it is to test these systems.

Testing the Chef: It's relatively easy to test if a chef makes up ingredients. You can ask them to write a recipe and check if the ingredients exist.
Testing the Search Party: It is incredibly difficult to test if the search party missed something. To know if they missed a note, you would have to read every single note in the patient's history yourself to see what was there. Since patient records can be thousands of pages long, no one has the time to do this for every AI test.

The authors note that current testing methods often only check if the AI found some good information, not if it found all the good information. This means we might think an AI is working perfectly when it's actually missing critical details.

What the Authors Are Saying (and Not Saying)

The paper is very careful not to say that hallucinations are solved or that we should stop worrying about them. They admit hallucinations are a "disturbing problem." However, they argue that we are currently obsessed with them while ignoring the bigger, harder-to-detect problem of missing information.

They don't claim to have a magic fix yet. In fact, they say there is no perfect solution for recall errors right now. The only way to be 100% sure an AI didn't miss anything is for a human to read the entire patient record, which defeats the purpose of using AI to save time. The paper suggests that we need new ways to test these systems, perhaps mixing human checks with smart automated tools, to ensure we aren't letting "silent omissions" slip through.

The Bottom Line

The main takeaway is a warning: Don't just watch out for the AI lying to you; watch out for the AI forgetting to tell you the truth.

As medical AI becomes more common, the authors believe the biggest hurdle won't be making the robot smarter; it will be making sure the robot's search party is thorough enough to find every single piece of the puzzle. If we don't fix the retrieval problem, the most advanced AI in the world could still miss a life-saving detail, and we might never even know it happened.

Technical Summary: Retrieval, not Hallucinations, as the Limiting Factor for LLM-based Clinical AI

Problem Statement
Current discourse surrounding errors in Large Language Model (LLM)-based clinical Artificial Intelligence (AI) predominantly focuses on precision errors, specifically "hallucinations" (Type I errors), where the model generates claims not substantiated by patient data. However, this paper argues that recall errors (Type II errors)—where the system fails to retrieve or include critical patient information present in the Electronic Health Record (EHR)—represent a more significant and under-addressed limitation.

In clinical AI workflows, such as ambient scribing or problem-oriented summarization, LLMs typically rely on Retrieval-Augmented Generation (RAG) to access patient history due to context length constraints. The core problem identified is that while precision errors can often be mitigated by linking claims to source documents, there is no equivalent, scalable mitigation strategy for recall errors. If a retrieval system fails to fetch a relevant note, or if the LLM fails to utilize fetched information, the resulting output lacks critical data. Because the "ground truth" of what should have been retrieved is often unknown in large, complex EHR datasets, these omissions are difficult to detect, potentially leading to "silent" clinical harm.

Methodology and Analytical Framework
This paper is a perspective article that synthesizes existing literature on LLMs, information retrieval (IR), and clinical AI evaluation. The authors employ the following analytical approaches:

Error Taxonomy: The authors map clinical AI errors onto the statistical framework of Type I (false positives/precision) and Type II (false negatives/recall) errors.
- Precision Errors: Defined as claims in the output not supported by patient data (e.g., hallucinations, confabulations).
- Recall Errors: Defined as the omission of important, data-supported information (e.g., missing a critical diagnosis, medication, or recent lab result).
Architectural Analysis: The paper analyzes the generalized RAG architecture (Query $\rightarrow$ Retrieval Engine $\rightarrow$ Prompt $\rightarrow$ LLM $\rightarrow$ Output) to identify fault sources. It posits that recall failures can originate from two points: the retrieval engine failing to return relevant documents, or the LLM failing to incorporate retrieved documents into the final generation.
Evaluation Benchmark Review: The authors review the state of evaluation metrics. They contrast text generation benchmarks (using metrics like ROUGE/BLEU or "nugget" evaluation) with information retrieval benchmarks (TREC-style pooling). They highlight that retrieval evaluation is inherently difficult because the "pooling" method (aggregating top results from multiple systems) cannot guarantee that all relevant documents are found, making it impossible to calculate exact recall performance on large datasets.

Key Contributions

Shift in Focus: The primary contribution is a call to shift the research and clinical focus from the well-studied problem of hallucinations to the under-studied problem of retrieval recall. The authors note that in qualitative interviews with clinicians, recall errors were initially overlooked until explicitly prompted, despite being recognized as a significant risk.
Identification of the "Silent Killer": The paper characterizes recall errors as a "silent killer" because, unlike hallucinations which can be flagged by source citation, missing information leaves no trace in the output, making it invisible to the end-user and difficult to audit.
Technological Disparity Analysis: The authors argue that performance in core information retrieval is improving at a slower pace than LLM capabilities. While LLMs are rapidly advancing, core retrieval engines often rely on established methods (e.g., BM25, dense vector retrieval) that lag behind. Consequently, as LLMs become more capable, the bottleneck for clinical AI will increasingly shift to the retrieval component's ability to find the right data.
Evaluation Critique: The paper critiques current evaluation paradigms, noting that standard metrics (like BLEU/ROUGE) and even "nugget" evaluation struggle to measure recall effectively in open-ended clinical tasks. It further argues that "LLM-as-a-judge" cannot serve as a gold standard for retrieval evaluation because retrieval and evaluation are fundamentally the same problem.

Results and Findings
As a perspective piece, the paper does not present new experimental results or quantitative data. Instead, its "findings" are derived from a synthesis of current trends:

Mitigation Asymmetry: Precision errors have established mitigation strategies (source linking), whereas recall errors lack a comparable, efficient solution that does not defeat the purpose of AI (saving time).
Retrieval Lag: Improvements in retrieval are likely to lag behind LLM improvements, meaning recall errors will become the dominant limiting factor as EHR data volumes grow.
Evaluation Feasibility: It is currently infeasible for every institution to perform rigorous, TREC-style recall evaluations for every clinical AI implementation due to the manual labor required and the lack of generalizable public datasets.

Significance and Claims
The paper claims that the limiting factor for the reliability and safety of future LLM-based clinical AI tools will not be the model's tendency to hallucinate, but its inability to retrieve the correct patient-level data.

The authors assert that while research into reducing hallucinations should continue, it must be amplified by research into recall evaluation and mitigation. They conclude that without addressing the fundamental difficulty of measuring and ensuring high recall in patient data retrieval, clinical AI tools risk causing patient harm through undetected omissions. The paper advocates for hybrid evaluation approaches that combine human assessment with automated methods to better identify and minimize recall errors in clinical deployments.