Active Evidence-Seeking and Diagnostic Reasoning in Large Language Models for Clinical Decision Support

Problem Statement

While Large Language Models (LLMs) have demonstrated strong performance on static medical benchmarks (e.g., MedQA, USMLE-style exams) where all patient data is provided simultaneously, clinical diagnosis in practice is a dynamic, iterative process. Real-world diagnosis relies on the hypothetico-deductive model, where clinicians must actively formulate hypotheses and seek evidence through multi-turn interactions (history taking, physical exams, lab tests) under conditions of initial information scarcity.

Current evaluation paradigms largely fail to capture this "active evidence-seeking" capability. Static, full-context benchmarks may overestimate model performance in interactive settings because they do not test a model's ability to decide what evidence to request, when to request it, or how to update hypotheses based on partial information. This gap creates a risk that models performing well on static tests may fail or hallucinate in realistic clinical decision support systems (CDSS) where information must be acquired incrementally.

Methodology: The ROUNDS-Bench Framework

To address this, the authors introduce ROUNDS-Bench, an OSCE (Objective Structured Clinical Examination)-inspired interactive framework designed to evaluate active diagnostic inquiry in a controlled, reproducible environment.

1. Standardized Patient Simulator (SP-sim):

• Core Mechanism: A high-fidelity simulator (powered by Qwen2.5-32B-Instruct) that mimics the information asymmetry of real doctor-patient interactions.
• Information Gating: The simulator starts with only basic demographics and the chief complaint. It releases subsequent evidence (symptom details, physical exam findings, lab results) only when the model submits logical, specific clinical requests (e.g., "Request Chest X-ray" or "Ask about family history").
• Constraints: The interaction is strictly controlled. Models cannot bypass the process with omniscient queries. The dialogue is limited to 10 turns, and the simulator never reveals the final diagnosis until the model explicitly terminates the session.
• Reproducibility: The system uses a deterministic temperature (0.0) and fixed seeds to ensure identical inputs yield identical outputs, avoiding the variability of open-ended chat evaluations.

2. Dataset Construction:

• Cohort: 468 standardized clinical cases covering six major clinical systems (Cardiovascular, Respiratory, Gastro-Hepatobiliary, Neurological, Infectious Diseases, Metabolic/Renal/Genitourinary).
• Stratification: Cases are strictly balanced (78 per system) and sourced from four heterogeneous corpora: MedQA (USMLE-style), MedMCQA (Indian exams), MedFound (MIMIC-III derived), and MedCaseReasoning (PMC case reports). This ensures diversity from textbook-style questions to complex, unstructured real-world narratives.

3. Dual-Task Evaluation Protocol:
The study employs a within-subject design to isolate the "Capability Gap":

• Task 1 (Full-Context): Models receive the complete electronic health record instantly. This establishes the theoretical upper bound of static diagnostic proficiency.
• Task 2 (Active Evidence-Seeking): Models start with only the chief complaint and must autonomously drive the multi-turn dialogue to acquire evidence before making a diagnosis.

4. Evaluation Metrics:

• Exact Diagnostic Accuracy: Strict semantic equivalence to the gold standard diagnosis.
• Strict Evidence Quality (StrictEQ): Measures whether the model's supporting evidence is fully grounded in the information actually retrieved during the interaction. It penalizes "Hallucinated Reasoning" (correct diagnosis without valid supporting evidence).
• Fully Supported Accuracy (FSA): The intersection of strict accuracy and strict evidence quality.

5. Models Evaluated:
15 state-of-the-art LLMs were tested, ranging from 7B to 32B+ parameters, including proprietary models (Gemini-2.5-Pro, GPT-4o-mini), open-weights models (Qwen series, DeepSeek series, Llama-3, GLM-4), and reasoning-enhanced variants (DeepSeek-R1-Distill).

Key Results

1. Significant Capability Gap:
Transitioning from passive (Task 1) to active (Task 2) diagnosis resulted in a consistent performance degradation across all 15 models:

• Diagnostic Accuracy: Decreased by an average of 12.75%.
• Evidence Quality: Decreased by an average of 24.36%.
• Error Distribution: Cases that were "Strictly Correct" in Task 1 often shifted directly to "Wrong" in Task 2, rather than becoming "Partially Correct." This indicates that under uncertainty, models do not become more conservative; instead, they construct erroneous causal chains based on insufficient evidence.

2. Evidence Quality vs. Diagnostic Accuracy (Hallucinated Reasoning):
The decline in evidence quality was often steeper than the decline in accuracy.

• Hallucinated Reasoning: Many models produced correct diagnoses without retrieving or citing the necessary discriminative evidence. For example, Gemini-2.5-Pro had a 49.36% accuracy in Task 2 but only 42.52% StrictEQ, implying ~7% of its "correct" diagnoses lacked evidentiary grounding.
• Safety Implication: This decoupling poses a safety risk for CDSS, as "lucky guesses" are not auditable and cannot be verified by clinicians.

3. Impact of Model Scale and Architecture:

• Scaling Laws: Larger models generally performed better, showing a log-linear improvement in Task 2 accuracy. However, even the largest models (e.g., Gemini-2.5-Pro, Qwen3-32B) failed to close the gap between static and active performance.
• Reasoning Architectures: Native reasoning models (e.g., Qwen3) showed robustness in active inquiry. However, distilled reasoning models at smaller scales (e.g., DeepSeek-R1-Distill-7B/14B) performed poorly in Task 2, suggesting that reasoning capabilities distilled from static Chain-of-Thought traces do not generalize well to proactive evidence seeking.

4. Domain and Data Source Variability:

• Resilient Domains: Models performed relatively well in Respiratory and Cardiovascular systems, likely due to standardized symptom-sign mappings.
• Vulnerable Domains: Significant fragility was observed in Neurological and Metabolic/Renal systems, which require nuanced physical exam localization or complex combinatorial lab interpretation.
• Data Source: Models performed significantly better on standardized exam-style data (MedQA/MedMCQA) than on real-world case reports (MedCaseReasoning). Performance on real-world data dropped to <35% for many models, highlighting a bottleneck in filtering noise and extracting signals from unstructured, high-entropy narratives.

Key Contributions

1. ROUNDS-Bench: A reproducible, controlled benchmark for active diagnostic inquiry that separates evidence acquisition from evidence synthesis, addressing the limitations of static full-context evaluations.
2. Quantification of the Capability Gap: The study provides empirical evidence that static benchmarks overestimate LLM performance in interactive settings, revealing a ~12.75% drop in accuracy and a ~24.36% drop in evidence quality.
3. Identification of "Hallucinated Reasoning": The introduction of the StrictEQ metric highlights a critical safety risk where models arrive at correct conclusions without valid, traceable evidence, undermining the auditability required for clinical trust.
4. Systematic Failure Analysis: The paper identifies specific failure modes, including premature diagnostic closure, inefficient questioning, and the inability of distilled reasoning models to generalize to active inquiry.

Significance and Claims

The authors claim that their work serves as a critical complement to existing static evaluations. They argue that high scores on licensing-style exams do not guarantee a model's ability to function safely in dynamic clinical workflows.

• Safety Warning: The paper emphasizes that for Clinical Decision Support Systems, "correct" answers derived from "unsubstantiated" reasoning are inherently untrustworthy. The observed gap suggests that current models may not be ready for autonomous deployment in settings requiring iterative evidence gathering.
• Evaluation Shift: The study motivates a transition in medical AI evaluation from static question-answering to authentic, dynamic interaction. It calls for future research to focus on improving interactive evidence seeking, auditability, and the ability to decide when enough information has been gathered.
• Modest Scope: The authors acknowledge limitations, noting that the simulation abstracts away sociopsychological factors, is text-only (lacking multimodal data like imaging), and relies on a single-reference diagnosis. They position ROUNDS-Bench as a necessary step toward safer, more clinically effective medical AI agents, rather than a final solution.