Technical Summary: Evaluating the Pre-consultation Ability of LLMs using Diagnostic Guidelines (EPAG)

Problem Statement

Large Language Models (LLMs) are increasingly integrated into clinical workflows, particularly for pre-consultation tasks such as history-taking and decision support. However, existing evaluations often focus on final diagnostic outcomes or multiple-choice question answering, failing to assess the critical intermediate process of information gathering. There is a lack of rigorous frameworks to evaluate whether LLMs can effectively elicit relevant patient information (History of Present Illness, or HPI) necessary for accurate diagnosis. Furthermore, the risks of erroneous outputs in clinical settings—such as misdiagnosis or incorrect prescriptions—necessitate specialized evaluation benchmarks that go beyond general medical knowledge.

Methodology

The authors introduce EPAG (Evaluating the Pre-consultation Ability of LLMs using diagnostic Guidelines), a benchmark dataset and automated evaluation pipeline designed to assess LLMs in a simulated pre-consultation setting.

Dataset Construction

The EPAG dataset comprises 520 synthetic patient profiles covering 26 diseases across 10 ICD-11 chapters and 10 primary specialties.

• Diagnostic Guidelines: A gold-standard set of diagnostic guidelines was constructed by expert clinicians from trusted sources (e.g., American College of Cardiology, NCCN). These guidelines specify key symptoms, ancillary symptoms, family history, and risk factors, assigned weights of "high" or "medium" significance.
• Filtering Criteria: Diseases were selected based on their ability to be reasonably diagnosed through consultation alone (excluding those requiring physical exams, X-rays, or MRIs) and their prevalence to ensure unbiased evaluation.
• Patient Profiles: Diverse profiles (age, sex, medical history) were generated using OpenAI o3-mini and qualitatively reviewed by clinicians to ensure realistic diversity and clinical detail.

Evaluation Pipeline

The evaluation simulates a dialogue between a patient-agent (acting out a specific profile) and a doctor-agent (the LLM under evaluation). The doctor-agent starts with basic information (age, sex, chief complaint) and iteratively asks questions with multiple-choice options to elicit HPI. The process is evaluated through two distinct tasks:

1. HPI-Diagnostic Guideline Comparison (Direct Evaluation):

   • Organization: The dialogue turns (Question, Options, Answer) are processed by an "organizer" model to extract atomic units of clinical information. Crucially, unselected options are treated as negative evidence (e.g., "Patient does not have symptom X").
   • Comparison: A "comparer" model matches each extracted unit against the pre-defined diagnostic guidelines.
   • Scoring: A unit receives a score of 1 (or 2 for high-significance guidelines) if it matches a guideline element; otherwise, it receives 0. Scores are calculated as both unweighted and weighted averages.

2. Disease Diagnosis (Indirect Evaluation):

   • An independent "diagnostician-agent" receives the full dialogue history and attempts to diagnose the patient.
   • Accuracy is measured by whether the predicted disease matches the gold label (including parent/child concepts in the taxonomy).

Experimental Setup

Eleven LLMs were evaluated, including proprietary models (GPT-4.1, Claude-3.7-Sonnet, etc.) and open-source models (Llama-3.2, Qwen2.5, Medgemma). Experiments were conducted over dialogue turns ranging from 5 to 9. A human expert baseline was established using the same pipeline.

Key Results

1. Model Size and Performance

Contrary to the intuition that larger models always perform better, the results showed that model size does not guarantee superior pre-consultation ability.

• While GPT-4.1 achieved the highest overall performance, smaller models like GPT-4o-mini outperformed their larger counterparts (GPT-4o) on HPI-guideline comparison scores.
• Open-source models generally underperformed proprietary models in their base forms.

2. Impact of Task-Specific Fine-Tuning

The study demonstrated that task-specific fine-tuning significantly bridges the performance gap.

• The authors fine-tuned Qwen2.5 models (7B, 32B, 72B) on a curated 3k pre-consultation dialogue dataset.
• Result: Fine-tuned smaller models (specifically the 7B and 32B variants) matched or even surpassed the performance of the strongest proprietary model (GPT-4.1) and the human baseline.
• The 32B model showed the most significant improvement, suggesting an optimal size for the specific dataset used.

3. HPI Quantity vs. Diagnostic Quality

The experiments revealed that increasing the amount of HPI does not necessarily improve diagnostic performance.

• As dialogue turns increased, the volume of collected HPI rose, but diagnostic accuracy did not correlate linearly.
• Qualitative analysis showed that models sometimes fixate on loosely connected keywords, generating guideline-related but clinically less significant questions, which can dilute focus or lead to misdiagnosis (e.g., prioritizing pain intensity over other symptoms leading to an incorrect diagnosis of Renal Colic instead of Acute Kidney Injury).

4. Language Influence

The language of the pre-consultation affects dialogue characteristics:

• Korean: Produced higher HPI-guideline comparison scores by casting a wider net and querying a broader array of symptoms.
• English: Achieved higher disease diagnosis accuracy by pursuing deeper, repetitive follow-ups on single symptoms, enhancing diagnostic confidence but generating fewer unique atomic units.

Significance and Contributions

The paper claims the following contributions to the field of clinical AI:

1. Framework Development: It establishes a systematic, two-tiered framework (HPI comparison and diagnosis) for evaluating the specific capability of LLMs in the pre-consultation phase, addressing a gap in existing literature that focuses primarily on final outcomes.
2. Dataset and Pipeline Open-Source: The authors open-sourced the dataset, diagnostic guidelines, and evaluation pipeline to facilitate reproducibility and further development of LLM applications in real-world clinical settings.
3. Empirical Insights: The study provides critical insights that challenge the assumption that larger models are inherently better for clinical tasks. It highlights that relatively small, open-source models, when fine-tuned on high-quality, task-specific data, can outperform larger, proprietary models.
4. Nuanced Evaluation: It demonstrates that the quality of information gathering (alignment with guidelines) is distinct from the quantity of information, and that the choice of language significantly shapes the diagnostic dialogue pattern.

The authors conclude that while their pipeline correlates well with human evaluation, it has limitations (e.g., excluding multimodal inputs like imaging) and should not be considered a definitive standard for all clinical applications without further assessment.