💻 computer science

Towards guideline-grounded clinical decision-making: a multi-scale study of retrieval augmented generation based large language model agents

This study demonstrates that integrating Retrieval-Augmented Generation (RAG) with clinical guidelines significantly enhances the diagnostic accuracy, treatment alignment, and transparency of large language model agents across diverse medical scenarios.

Original authors: Romain André, John Hermans, J.P. Sedelaar, Henkjan Huisman

Published 2026-10-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Romain André, John Hermans, J.P. Sedelaar, Henkjan Huisman

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern hospital, a doctor's mind is a powerful engine for diagnosis, but it is not infallible. It must sift through a chaotic stream of patient history, lab results, and imaging scans, often under time pressure, to find the correct answer. In recent years, a new type of artificial intelligence called a large language model has emerged, capable of reading vast amounts of medical text and answering complex questions. These models are trained on millions of documents, effectively memorizing a library of medical knowledge. However, this memory has a flaw: the model cannot always distinguish between what it has learned and what is true for a specific patient right now. It might confidently state a fact that is outdated, or worse, invent a detail that never happened. To fix this, researchers have developed a method called retrieval-augmented generation. Instead of relying solely on its internal memory, the system pauses to look up the most relevant, up-to-date guidelines from a trusted database before it speaks. It is like a doctor who, before making a final decision, quickly checks the latest official protocol to ensure their reasoning is grounded in current evidence.

A team of researchers at Radboud University Medical Center in the Netherlands set out to test whether this "look-before-you-speak" approach could actually improve the way artificial intelligence makes clinical decisions. They did not just ask the AI a simple question like "What is the treatment for appendicitis?" Instead, they created a complex simulation that mimics the real, messy process of diagnosing a patient in an emergency room. Using data from 2,400 real patient cases involving acute abdominal pain, they tasked an AI agent with acting as a doctor. The agent started with only the patient's initial story and had to decide, step by step, which physical exams to perform, which blood tests to order, and which imaging scans to request. After each test, the agent received the results and had to decide what to do next, eventually stopping when it felt it had enough information to name the disease and prescribe a treatment plan. The researchers tested eight different versions of the AI, ranging from smaller, faster models to massive, highly complex ones, and compared how they performed with and without the ability to look up guidelines during the process.

The results showed that giving the AI access to a curated library of medical guidelines significantly improved its performance. When the models were allowed to retrieve and read relevant guidelines at every step of the decision-making process, their diagnostic accuracy rose from an average of 55.64 percent to 64.29 percent. This improvement was not limited to the largest or most expensive models; it helped every single version of the AI tested, regardless of its size or how much medical knowledge it had been pre-trained on. The system became better at ordering the correct tests, such as requesting a specific enzyme test for pancreatitis or an ultrasound for gallbladder issues, and it was more likely to arrive at the correct final diagnosis and treatment plan. The study found that the AI was not just guessing better; it was following a more logical path, asking for the right information in the right order, much like a skilled clinician would.

However, the researchers discovered that simply adding a library of text to the AI was not enough to guarantee success. The way the system searched for information mattered just as much as the information itself. The most critical factor was a step where the AI was forced to rewrite the entire patient story into a short, focused search query before looking up the guidelines. Without this step, the AI would try to search using the entire, messy history of the patient, which often confused the search engine and returned irrelevant documents. By compressing the patient's story into a clear, specific question, the AI could find the exact guideline it needed. Furthermore, the researchers found that the timing of the search was important. If the AI tried to look up guidelines before it had even performed a single physical exam or lab test, the results were less helpful. The system worked best when it waited until it had gathered some initial facts before consulting the library.

The study also revealed that the quality of the guidelines themselves played a major role. When the researchers added extra documents to the library that covered diseases with similar symptoms but were not the target of the specific case, the AI's performance dipped slightly, though it still remained better than if it had no library at all. This suggests that while the system is robust, it is sensitive to "noise" or confusing information. The researchers concluded that for artificial intelligence to be truly useful in a hospital, it cannot simply be a smart chatbot that knows a lot. It must be a structured agent that knows how to find the right evidence at the right time, rewrite its own questions to stay focused, and verify its steps against trusted sources. While this technology shows great promise for improving diagnostic accuracy and making the decision-making process more transparent, the authors caution that it is not yet ready to replace human doctors. The system still needs human oversight, and future studies will need to prove that it can safely handle the full pressure and complexity of a real emergency department. For now, the work demonstrates that when artificial intelligence is grounded in verified, external knowledge, it becomes a more reliable partner in the difficult task of saving lives.

Technical Summary: Towards Guideline-Grounded Clinical Decision-Making

Problem Statement
Large Language Models (LLMs) show promise for clinical decision support but are currently limited by opaque knowledge sources, inconsistent reasoning, and the generation of hallucinated information. In sequential diagnostic pathways, these limitations undermine reliability, affecting not only the final diagnosis but also the sequence of examinations requested and the timing of diagnostic commitment. While Retrieval-Augmented Generation (RAG) offers a mechanism to ground models in external, authoritative sources without retraining, existing evaluations often focus on single-turn question answering or licensing-style exams. These do not capture the iterative, information-gathering nature of real-world clinical decision-making, where agents must autonomously decide which tests to order and when to stop gathering evidence. Furthermore, RAG performance is highly sensitive to corpus preparation, query formulation, and evidence presentation, yet the specific design elements required for clinical utility remain under-explored.

Methodology
The authors extended the MIMIC-IV-Ext Clinical Decision-Making (MIMIC-CDM) benchmark to evaluate RAG in an iterative, multi-step diagnostic setting. The study utilized 2,400 real patient pathways from emergency departments presenting with acute abdominal pain (appendicitis, cholecystitis, diverticulitis, or pancreatitis).

Experimental Setup: The framework required LLM agents to autonomously request physical examinations, laboratory tests, and imaging modalities based on the patient's history. The agent iteratively integrated new findings until it judged sufficient evidence existed to deliver a final diagnosis and care plan.
RAG Integration: In the RAG condition, at each decision step, the accumulated patient history was autonomously condensed into a focused search query. Relevant guideline passages were retrieved from a curated corpus, reranked, and appended to the model's context before the next action was generated.
Model Scope: Eight locally deployable LLM backbones were evaluated, ranging from 1B to 70B parameters, including general instruction-tuned, biomedical-fine-tuned, and clinical foundation models. All models were run locally under matched 4-bit quantized inference settings to comply with data governance constraints.
Ablation and Corpus Analysis: The study systematically varied retrieval design components (query rewriting, reranking, evidence ordering, retrieval timing) and corpus composition (target pathology coverage, document cleaning, and diagnostic ambiguity) to isolate factors driving performance.

Key Contributions

Iterative Clinical Benchmarking: The study moves beyond static QA to evaluate RAG in a dynamic, sequential decision-making environment where the agent controls the diagnostic workflow.
Systematic Ablation of RAG Components: The authors demonstrate that the clinical benefit of RAG is not inherent to the mere addition of retrieved text but depends critically on specific retrieval design choices.
Corpus Sensitivity Analysis: The research quantifies how retrieval fidelity and downstream clinical performance are affected by corpus scope, document preparation (cleaning/OCR correction), and the introduction of diagnostic ambiguity (distractors).

Results

Diagnostic Accuracy: Across all eight backbones, RAG consistently improved mean macro-averaged diagnostic accuracy from 55.64% (non-RAG) to 64.29% (RAG), an absolute gain of 8.65 percentage points ( $P=0.0039$ ). This improvement was observed across model scales (1B to 70B) and both general and medically specialized models.
Pathology-Specific Gains: Improvements varied by condition, with the largest gains observed for diverticulitis (+17.33 points) and pancreatitis (+9.86 points), while appendicitis showed a marginal, non-significant improvement, likely due to higher baseline performance.
Process-Level Alignment: RAG improved alignment with clinician test-ordering (physical exam, imaging, labs) by 5.17 points and treatment alignment by 3.97 points in a matched three-backbone analysis.
Critical Design Factors:
- Autonomous Query Rewriting: This was the most impactful component. Removing query rewriting (using raw context as the query) reduced retrieval precision at 6 (P@6) by 11.12 points and diagnostic accuracy by 2.73 points.
- Reranking: Removing reranking degraded retrieval fidelity and diagnostic accuracy.
- Corpus Composition: Expanding the corpus from 4 to 16 guidelines improved performance. However, adding 16 "ambiguous" guidelines (non-target abdominal pain causes) reduced retrieval fidelity and diagnostic accuracy, though RAG still outperformed the non-RAG baseline.
- Retrieval Timing: Initiating retrieval before the first clinical observation (history only) significantly reduced test-ordering alignment, suggesting that premature retrieval without sufficient clinical data can bias the diagnostic pathway.

Significance and Claims
The paper concludes that guideline-grounded RAG can improve both the diagnostic outcomes and the clinical decision pathways of LLM agents while increasing traceability and transparency. However, the authors emphasize that the magnitude of this benefit is contingent upon structured retrieval design, particularly autonomous task-focused query rewriting and passage-level reranking.

The study explicitly states that while RAG improves diagnostic accuracy and process alignment, it does not resolve all workflow or safety barriers. A model may reach the correct diagnosis despite an incomplete or suboptimal work-up; therefore, endpoint accuracy alone is not evidence of clinical safety. The authors assert that "RAG" is not a sufficiently precise design description unless corpus construction, query generation, and evidence organization are reported. They conclude that these findings support the evaluation of RAG as supervised, auditable clinical decision support but do not support its autonomous clinical deployment without further prospective evaluation under clinician oversight.

More like this