Technical Summary: Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries

1. Problem Statement

The rapid adoption of AI tools by physicians for clinical decision support has outpaced the development of rigorous evaluation frameworks. Current medical AI benchmarks rely heavily on:

• Hypothetical or Exam-Style Questions: Datasets often derived from licensing exams or co-designed by AI agents, which fail to capture the complexity, distribution, and nuance of real-world queries asked at the point of care.
• Inadequate Grading Methods: Evaluations frequently utilize fixed rubrics or a small number of generalist physicians to rate questions across diverse domains, ignoring the specialization that defines modern medicine.
• Reliance on LLM Judges: There is an emerging debate on replacing human experts with Large Language Models (LLMs) as judges, a practice that remains largely untested for high-stakes clinical reasoning.

The authors argue that these gaps necessitate an evaluation grounded in the realities of clinical practice, utilizing real-world queries and judgment from appropriately matched domain experts.

2. Methodology

The study, titled Real-POCQi (Real-world Point-Of-Care Queries), employed a randomized, blinded, pairwise evaluation design.

Data Collection and Benchmark Construction

• Query Source: The primary dataset consists of 620 questions sampled from over one million daily queries submitted by physicians to the OpenEvidence (OE) platform. These questions span 30 medical specialties (e.g., cardiology, gastroenterology, nephrology) and cover topics from diagnosis to drug safety.
• Sensitivity Dataset: 187 questions from the HealthBench benchmark were included for sensitivity analysis.
• Preprocessing: Questions were de-identified and lightly rewritten to remove Protected Health Information (PHI) while retaining clinical content.

Evaluation Design

• Participants: 149 practicing, licensed U.S. physicians across 36 states were recruited from the IQVIA OneKey database.
• Matching: Evaluators were matched to questions based on their clinical specialty to ensure domain expertise.
• Models Compared:
  • Specialized Tool: OpenEvidence (OE).
  • General-Purpose Frontier Models: GPT-5.5, Gemini 3.1 Pro, and Claude Opus 4.8.
• Procedure:
  • Physicians performed head-to-head comparisons of two AI-generated answers for a given query.
  • Blinding: Vendor identities were hidden. Answers were rendered identically (text-only or text + citations) to prevent identification by style.
  • Randomization: A nested randomization design assigned rendering modes (2:1 split for text-only vs. text+citation), question pairs, and answer positions (A/B).
  • Dimensions: Evaluators rated pairs on five axes using a 5-point Likert scale: Accuracy, Clinical Utility, Source Quality, Verifiability, and Completeness.

Statistical Analysis

• Primary Metric: One-vs-Rest (OvR) Win Difference, defined as $(Wins - Losses) / N$. This measures the net preference rate of a system against all others.
• Secondary Metric: Pairwise win-rate matrices ($P(A \succ B)$).
• Robustness: Sensitivity analyses were conducted stratifying by answer length, citation display, evaluator OE-user status, and dataset source (Real-POCQi vs. HealthBench).
• LLM-as-Judge Comparison: The three frontier models were also prompted to act as judges on the same data to compare their rankings against human experts.

3. Key Results

Performance of Specialized vs. General Models

• Dominance of Specialized Tool: OpenEvidence (OE) achieved positive win differences across all five evaluation dimensions in the primary analysis (text-only).
  • Accuracy: OE win difference = 24.7% ($p < 0.001$). GPT-5.5 was significantly lower at -21.1%.
  • Clinical Utility: OE win difference = 29.6% ($p < 0.001$).
  • Source Quality: OE achieved the highest win difference of 38.8% ($p < 0.001$).
• General-Purpose Models: GPT-5.5 consistently showed the lowest win differences (often negative) across all axes. Gemini 3.1 Pro and Claude Opus 4.8 performed near parity with each other but significantly worse than OE.
• Sensitivity Analyses: Results remained consistent when stratifying by:
  • Citation Display: Trends held in the text+citation condition.
  • Answer Length: OE's advantage persisted regardless of relative answer length, though accuracy ratings were sensitive to length generally.
  • Evaluator Status: OE outperformed general models for both existing OE users and non-users, though the effect size was slightly attenuated for non-users.
  • Dataset Source: On the independent HealthBench dataset, OE maintained positive win differences across all axes, though effect sizes were slightly smaller.

Human vs. LLM Judges

• Divergence in Rankings: While LLM judges generally agreed with human experts on which model was best (OE), they significantly disagreed on the ranking of the other models.
• Self-Preference: GPT-5.5 exhibited high self-preference when acting as a judge, leading to a negative correlation (Kendall's $\tau = -0.200$) with human expert ratings.
• Overconfidence: LLM judges reported more extreme win differences (overconfident) compared to human experts.
• Correlation: Correlation between LLM judges and human experts ranged from 0.267 (Claude) to 0.467 (Gemini), indicating substantial misalignment in nuanced clinical judgment.

4. Key Contributions

1. Real-POCQi Benchmark: The release of the largest public benchmark of real-world clinical queries (620 questions, 30 specialties) submitted by practicing clinicians, accompanied by 1,156 expert ratings.
2. Specialty-Matched Evaluation: A rigorous evaluation framework that matches physician graders to specific clinical topics, addressing the limitation of generalist grading in specialized domains.
3. Empirical Evidence on Specialization: Demonstrated that targeted engineering and customization (as seen in OE) yield meaningful gains in performance and utility over frontier general-purpose models for point-of-care decision support.
4. LLM Judge Limitations: Provided large-scale evidence that LLMs, even frontier models, cannot yet reliably substitute for human experts in grading clinical reasoning, particularly regarding ranking and self-preference biases.

5. Significance and Claims

The paper claims that the consistent advantage of the specialized tool (OE) over general-purpose models does not imply that general models are incapable of answering clinical questions, but rather that targeted engineering and customization are essential for high-performance clinical decision support.

The authors emphasize that:

• Evaluation Must Reflect Reality: AI tool evaluations must move beyond standardized, hypothetical benchmarks to reflect real-world query distributions and utilize expert judges who mirror the specialization of modern medicine.
• Caution with Automation: The systematic differences between human and LLM judges highlight the risks of automating clinical judgment without substantial tuning and validation.
• Context Matters: The performance of AI tools is highly dependent on the deployment setting; a tool optimized for point-of-care queries may not generalize to other environments, and vice versa.

The study concludes by releasing the dataset and analysis code to advance the science of clinical AI evaluation, aiming to shift the field toward measuring the net impact of these tools on clinical judgment and patient outcomes.