Teaching LLMs How ICU Physicians Approach Clinical Reasoning Through OMOP-Aligned Retrieval Improves Reasoning Across Clinical Domains

Technical Summary: Teaching LLMs How ICU Physicians Approach Clinical Reasoning Through OMOP-Aligned Retrieval

Problem Statement
Clinical decision-making in the Intensive Care Unit (ICU) requires synthesizing vast, heterogeneous data from Electronic Health Records (EHRs) to form hypotheses and treatment plans. While Large Language Models (LLMs) show promise in interacting with EHRs, existing applications predominantly focus on surface-level retrieval or factual recall. They often fail to replicate the inductive and deductive reasoning clinicians employ to select decision-relevant evidence and reason over it. Furthermore, current datasets and benchmarks largely assess final answers via multiple-choice questions or isolated retrieval tasks, offering limited supervision on how clinicians identify, prioritize, and contextualize patient information. There is a critical gap in training resources that teach models to perform multi-stage, context-aware clinical reasoning rather than simple information extraction.

Methodology
The authors introduce ICU-REACT (Intensive Care Unit Reasoning for Electronic Health Record-Anchored Decision Support Tasks), a clinician-supervised dataset, and Clin-REACT, a family of fine-tuned LLMs.

1. Dataset Creation (ICU-REACT):

   • Seed Generation: A multi-agent framework using GPT-4.1 generated initial decision-making questions and retrieval tasks based on nine ICU domains (e.g., sepsis, hemodynamic instability).
   • Clinician-in-the-Loop: 19 clinicians across multiple specialties reviewed, refined, and validated the seed samples via a custom annotation tool. They verified clinical relevance, corrected variable selection, and evaluated reasoning logic.
   • Standardization: All clinical variables were mapped to the OMOP Common Data Model to ensure interoperability across EHR systems.
   • Augmentation: The clinician-validated seed set (213 samples) was split into training (142) and test (71) sets. The training seed was augmented using a self-instruct pipeline involving GPT-5.2 and GPT-4o-mini. This process generated new context-question pairs and, crucially, employed a "reasoning refinement" strategy where an initial imperfect answer was generated and then improved by a refinement agent using clinician feedback as a guide.
   • Final Datasets: Three training sets were created (Small: 10k, Medium: 5.3k, Large: 27.9k samples) tailored to different model capacities.

2. Model Training (Clin-REACT):

   • Backbones: Four open-source models were used: Llama 3.1 8B, Llama 3.3 70B, Gemma 4 31B, and Baichuan M1 14B.
   • Fine-Tuning: Models were trained using Supervised Fine-Tuning (SFT) with Low-Rank Adaptation (LoRA). The objective was to teach the model to take a patient context, a decision question, and an initial reasoning draft, and output a refined reasoning explanation and improved answer. This "reasoning refinement" approach was chosen to expose models to failure modes and correction steps.

3. Evaluation:

   • Internal: Performance was tested on the held-out ICU-REACT test set (71 samples) using automated retrieval metrics (F1, precision, recall) and LLM-judge scoring based on clinician-derived rubrics.
   • External: Models were evaluated on four independent benchmarks: SCT-Bench (script concordance under uncertainty), ER-Reason (emergency medicine), MedRBench (general clinical reasoning), and VivaBench (multi-turn diagnostic interviews).
   • Baselines: Comparisons were made against open-source general-purpose LLMs (e.g., GPT-OSS, Gemma, Llama) and open-source medical LLMs (e.g., Meditron, MedGemma, HuatuoGPT).

Key Results

• Superior Performance: Clin-REACT models consistently outperformed their untrained backbone models and competitive open-source general-purpose and medical LLMs across all five benchmarks.
• Transferability: Training on ICU-specific reasoning improved performance on external benchmarks spanning emergency medicine and general clinical reasoning. For instance, Clin-REACT 31B achieved the highest macro-average score (50.4) across all benchmarks, outperforming larger models like GPT-OSS 120B.
• Downstream Impact: Improvements were not limited to information retrieval. Clin-REACT models showed significant gains in downstream tasks, including differential diagnosis, treatment planning, and assessment recommendations. Strong correlations were observed between information retrieval scores and downstream diagnosis/treatment accuracy (ρ = 0.85).
• Safety and Faithfulness: Clin-REACT models demonstrated higher task faithfulness (96.7%–99.8%) and safety scores (78.6%–93.9%) compared to general-purpose and medical baselines.
• Efficiency: Lightweight LoRA fine-tuning allowed smaller models (e.g., Clin-REACT 14B) to match or exceed the performance of much larger specialized medical models (e.g., MedGemma 27B).

Significance and Claims
The paper claims that expert reasoning supervision in a high-acuity, data-dense environment (ICU) can teach LLMs clinical reasoning skills that generalize beyond the training domain.

• Beyond Retrieval: The study demonstrates that teaching models how to reason—specifically by identifying relevant evidence and refining hypotheses—yields transferable skills that improve performance on diverse clinical tasks, not just ICU-specific retrieval.
• Quality over Scale: The results suggest that the quality of reasoning supervision (via clinician-curated refinement) is a more potent driver of performance than model size alone, as smaller fine-tuned models outperformed larger, pre-trained medical models.
• Benchmark Validity: The strong correlation between ICU-REACT performance and external benchmarks (SCT-Bench, ER-Reason, VivaBench) validates ICU-REACT as a measure of broad clinical reasoning ability, not just domain-specific knowledge.
• Limitations and Future Work: The authors acknowledge limitations, including the reliance on LLM-as-a-judge for some evaluations, the relatively small test set size, and the lack of prospective clinical validation. They emphasize that while the models show improved reasoning, real-world clinical utility and safety require prospective evaluation with practicing clinicians before deployment. The work provides a framework for training models to identify which evidence matters, a critical step toward reliable clinical AI.