Technical Summary: Evaluating Clinical Competencies of Large Language Models with a General Practice Benchmark

Problem Statement

While Large Language Models (LLMs) have demonstrated significant potential in healthcare, particularly in specialized tasks and medical knowledge recall, their ability to function as autonomous General Practitioners (GPs) remains unproven. Existing evaluation frameworks predominantly rely on exam-style, multiple-choice questions (MCQs) or simplified question-answer formats. These benchmarks fail to capture the holistic, open-ended, and continuous nature of real-world general practice, which requires symptom-oriented differential diagnosis, management of multimorbidity, triage under uncertainty, and the application of the biopsychosocial model. Consequently, there is a lack of competency-based structures aligned with the actual duties of GPs, leaving the reliability of LLMs in clinical general practice uncertain.

Methodology

To address this gap, the authors propose GPBench, a novel evaluation framework and dataset designed to assess LLMs through the lens of GP competency.

1. Competency Model

Derived from real-world clinical practice and established frameworks (WONCA, ACGME, Iceberg Model, Onion Model) via a Delphi expert consensus, the model defines six primary indicators and fourteen secondary indicators:

• Basic Medical Knowledge (10%)
• Diagnosis (40%): Including differential diagnosis, medical history taking, complication identification, acute/critical condition recognition, and referral decision-making.
• Decision-making (30%): Including treatment plan formulation, adverse drug reaction management, contraindication awareness, and alternative therapy selection.
• Health Management (10%): Including health education and tertiary prevention.
• Health Economics (5%): Cost-effective care coordination.
• Medical Ethics and Humanities (5%): Humanistic care competency.

2. Evaluation Dataset

The dataset comprises three distinct test sets, all constructed using anonymized real-world outpatient records from Tertiary A-grade hospitals in China and open-source data, with all interactions conducted in Chinese:

• MCQ Test Set: 3,661 questions evaluating foundational knowledge and theoretical understanding.
• Clinical Case Test Set: 70 open-ended cases covering 8 major chronic diseases and 10 common symptoms. LLMs must generate systematic diagnostic and treatment plans based on medical records.
• AI Patient Test Set: 70 interactive simulations where an LLM acts as a doctor interviewing an AI-generated patient to gather medical history.

3. Evaluation Protocol

• Models Evaluated: Ten state-of-the-art LLMs, including general-purpose models (e.g., GPT-4o, Claude-3.5-Sonnet), reasoning-optimized models (e.g., o1-preview, DeepSeek-R1), and medical-specialist models (e.g., HuatuoGPT-o1-7B).
• Scoring:
  • MCQ: Accuracy calculated via majority voting over five inference runs.
  • Clinical Cases & AI Patient: A panel of three licensed GPs (with 10–15 years of experience) scored responses based on detailed, expert-annotated criteria.
• Human Benchmark: Six GPs stratified by experience (<5, 5–10, >10 years) were evaluated on the same datasets to provide a performance baseline.

Key Results

The evaluation of ten LLMs yielded the following findings:

1. Performance vs. Human Physicians

• Knowledge vs. Reasoning: LLMs generally outperformed human physicians on the MCQ test set (factual recall). However, on the Clinical Case and AI Patient test sets (reasoning and interaction), LLMs significantly underperformed compared to even the least experienced human GPs (Group C).
• Best Performer: DeepSeek-R1 achieved the highest scores among LLMs, reaching a competency level comparable to Group C human physicians in clinical cases, but still lagging in history taking.

2. Specific Competency Deficiencies

• Diagnostic Reasoning: Most models failed to construct complete chains of diagnostic reasoning. In the Clinical Case set, 7/10 models scored below 70 on differential diagnosis.
• Medical History Taking: LLMs exhibited severe deficiencies in the AI Patient test set. Even the best model (o1-preview) scored only 55.20, failing to systematically explore critical symptoms (e.g., pain characteristics, associated symptoms) or gather comprehensive personal/family histories.
• Treatment Planning: Models struggled to formulate optimal treatment plans. Only DeepSeek-R1 scored above 70. Common failures included:
  • Lack of Treatment Goals: Failure to specify targets for chronic diseases (e.g., blood pressure/glucose control).
  • Pharmacotherapy Risks: Omission of core drugs, neglect of drug interactions/contraindications, and incorrect dosages.
  • Hallucinations: Arbitrary grading and staging of diseases without clinical evidence (e.g., "Grade 0" appendicitis).
  • Blind Spots: Failure to identify comorbidities and complications, particularly in metabolic and cardiovascular contexts.
• Health Management: While LLMs performed well in structuring health education, their advice often remained at a broad, principled level lacking actionable, specific guidance (e.g., failing to specify sodium limits).

3. Model Comparisons

• Reasoning Models: Reasoning-optimized models (DeepSeek-R1, o1-preview) generally outperformed general-purpose models in diagnostic and treatment tasks.
• Parameter Scale: Larger parameter models (e.g., Qwen2.5-72B) consistently outperformed their smaller counterparts (Qwen2.5-7B).
• Specialist Models: Medical-specialist models (HuatuoGPT-o1-7B) showed strong performance in specific areas like health economics and ethics but did not surpass reasoning-optimized general models in core diagnostic reasoning.

Significance and Claims

The paper claims that current LLMs are not suitable for autonomous deployment in clinical general practice. The study highlights that while LLMs excel at static knowledge retrieval, they lack the practical competency, interactive reasoning, and holistic understanding required for real-world GP duties.

Key contributions include:

1. GPBench Framework: A competency-based evaluation model that moves beyond exam-style testing to assess the full spectrum of GP responsibilities.
2. Granular Deficiency Analysis: Identification of specific failure modes, such as hallucination in disease staging, omission of core drugs, and inability to conduct systematic history taking.
3. Realistic Benchmarking: The use of real-world clinical records and interactive simulations provides a more rigorous assessment of LLM readiness for clinical integration than previous benchmarks.

The authors conclude that continuous human oversight is mandatory for any realistic application of LLMs in general practice. They recommend that future research focus on enhancing systematic medical knowledge, integrating structured clinical workflow data to reduce hallucinations, and improving modeling of human physiology and disease progression. The dataset is made publicly available to support further development and optimization of clinical decision support systems.