Technical Summary: Large Language Models Improve Physician Accuracy but Lead to False Reliance

Problem Statement

While Retrieval-Augmented Generation (RAG) systems promise to mitigate the hallucination and transparency issues of Large Language Models (LLMs) in clinical settings by providing source-linked evidence, their actual impact on physician decision-making remains untested. Existing research has largely operated in parallel: one stream evaluates LLM efficacy in enhancing clinical decisions, while another assesses the accuracy of retrieval-based outputs. A critical gap exists in understanding the interaction between a physician, the LLM's generated answer, and the perceived validity of the accompanying citations. Specifically, it is unknown whether source-linked assistance improves physician accuracy or if the presence of citations creates a "grounding-dependent safety risk," where physicians defer to incorrect advice simply because it appears supported by evidence.

Methodology

System Development: CORA

The authors developed CORA (Citation-Oriented Retrieval Assistant), an agentic RAG system designed for dermatological decision support.

• Architecture: CORA employs an iterative retrieval loop orchestrated by a Qwen3 model. For each clinical query, it plans a retrieval strategy, searches a vector database containing clinical guidelines (EADV), textbooks (Fitzpatrick's), and case reports, and evaluates evidence sufficiency. If evidence is insufficient, it reformulates the query and repeats retrieval (up to three iterations).
• Output: The system generates an answer using a generation LLM, strictly constrained by the retrieved context, and outputs the answer with linked citations.
• Knowledge Base: The system indexes 72 EADV guidelines, the 9th edition of Fitzpatrick's Dermatology in General Medicine, and 7,936 dermatology case reports (2004–2024).

Datasets

Two distinct datasets were constructed to evaluate model performance and physician interaction:

1. DermBenchQA: A dataset of 4,855 single-best-answer multiple-choice questions derived from four medical benchmarks (MedMCQA, MedQA, MMLU, IMPP).
2. DermCaseQA: A contamination-resistant dataset of 998 open-ended questions generated from 893 dermatology case reports published in 2025 or later (post-dating the training cutoffs of all evaluated models).

Experimental Design

The study involved a two-stage evaluation:

1. Benchmark Evaluation: CORA was compared against five base LLMs (GPT-5, Llama-4, Qwen 2.5, Mistral Large 2, Gemma 3) without retrieval augmentation. Comparisons were restricted to cases where the retriever reached "sufficiency" within three iterations to ensure a fair comparison on identical items.
2. Reader Study: A within-subjects study involving 46 physicians from 21 countries.
   • Procedure: Physicians answered 16 dermatology questions unaided. They were then shown CORA's answer and its cited sources. They were instructed to rate whether each citation supported the answer (independent of their belief in the answer's correctness) and then decide whether to retain or revise their initial response.
   • Analysis: The study measured changes in accuracy, the correlation between perceived citation support and answer correctness, and the "appropriateness of reliance" (adopting correct advice vs. resisting incorrect advice).

Key Results

1. Model Performance

• Non-Inferiority: Across all five backbone models, CORA maintained non-inferior performance compared to the baseline (no retrieval) on the DermBenchQA set.
• Gains on New Data: On the contamination-resistant DermCaseQA set (post-training cutoff), retrieval provided substantial gains. For example, Gemma 3 improved by 22.7 percentage points (pp), and GPT-5 improved by 3.4 pp.
• Inverse Scaling: The magnitude of improvement scaled inversely with baseline capability; weaker models saw larger gains, particularly in rare or complex disease categories (e.g., genetic/rare, metabolic/systemic).

2. Physician Accuracy

• Overall Improvement: Physician accuracy increased from 70.8% (unaided) to 82.6% (with CORA assistance).
• Correction vs. Error: Assistance corrected 110 initially incorrect responses but overturned 23 initially correct responses into incorrect ones.

3. The Role of Citation Support

• Predictive Power: Perceived citation support strongly predicted correctness. When physicians judged at least one citation as supportive, the LLM's answer was correct 92.9% of the time (vs. 65.8% when unsupported).
• Physician Correction: Physicians were significantly more likely to correct an initial error when they perceived citation support (64.6% success rate) compared to when they did not (23.9%).

4. The "Grounding-Miscalibration" Risk

The study identified a critical asymmetry in physician reliance based on perceived citation support:

• Adoption of Correct Advice: When CORA was correct, physicians adopted the advice in 76.9% of cases where citations were perceived as supportive, compared to only 34.0% when citations were not perceived as supportive.
• Resistance to Incorrect Advice: When CORA was incorrect, physicians resisted the advice in 92.0% of cases where citations were not perceived as supportive. However, this resistance dropped precipitously to 34.8% when physicians judged the citations as supportive.
• Conclusion: The same signal (perceived citation support) that drives appropriate reliance on correct advice also drives harmful deference to incorrect advice. This phenomenon is termed "grounding-miscalibration."

Significance and Claims

The paper claims that while retrieval-augmented LLMs can significantly improve physician accuracy, their safety profile is contingent on the alignment between the appearance of evidentiary support and the actual correctness of the answer.

• Beyond Aggregate Accuracy: The authors argue that aggregate accuracy is an inadequate endpoint for clinical AI. A system can raise mean accuracy while simultaneously increasing the rate of "harmful reversals" (abandoning correct answers for incorrect ones) if the system generates incorrect answers that appear well-grounded.
• The Citation Paradox: The presence of citations, intended to enable verification, can paradoxically substitute for it. Physicians in the study treated the presence of a citation as a proxy for correctness, leading them to abandon their own correct assessments when the LLM provided a citation-supported (but factually wrong) answer.
• Implications for Development: The authors conclude that future clinical LLMs must be evaluated not just on answer accuracy, but on how the presentation of evidence shapes reliance on both correct and incorrect advice. They suggest that systems need to explicitly flag when retrieval is weak or when a source matches a topic but does not substantiate a specific claim, to prevent the "grounding-miscalibration" failure mode.

The study does not claim that citations are inherently bad, but rather that the perception of support by clinicians is a double-edged sword that current evaluation metrics fail to capture.