Ten Headache Specialists versus Artificial Intelligence for Clinical Literature Summarization: A Critical Evaluation and Comparison

Technical Summary: Ten Headache Specialists versus Artificial Intelligence for Clinical Literature Summarization

Problem Statement
The exponential growth of medical literature presents a significant challenge for evidence-based medicine, where clinicians must synthesize vast amounts of data within time-constrained patient visits. While Retrieval-Augmented Generation (RAG) using Large Language Models (LLMs) has shown promise in summarizing clinical evidence, there is a critical gap in understanding how these AI-generated summaries compare to those written by clinical experts. Specifically, it remains unclear what characteristics distinguish expert-authored synthesis from LLM outputs, what attributes experts prioritize beyond standard quantitative metrics, and whether current RAG systems can reliably replicate the nuance, clinical utility, and reference quality of human experts.

Methodology
The study employed a multidimensional evaluation framework comparing summaries generated by three state-of-the-art LLMs (Anthropic's Sonnet, OpenAI's GPT-4o, and Meta's Llama 3.1) against summaries written by ten academic headache specialists.

• Framework Construction: The authors utilized an agentic AI framework (clinfo.ai) based on a chain-of-LLMs pipeline. This system performed five sequential steps: (1) Query Generation using MeSH terms, (2) Information Retrieval from PubMed, (3) Relevance Classification via an LLM, (4) Summarization of relevant abstracts, and (5) Synthesis into a structured, evidence-based summary. The pipeline was optimized with a headache specialist to prioritize high-level evidence, original research over narrative reviews, and clinically actionable details (e.g., dosages).
• Dataset Generation: Ten headache specialists (average 21.9 years of experience) formulated 13 clinical questions. One specialist wrote an expert summary for each of 10 evaluation questions. The AI framework generated three corresponding summaries (one per model) for each question, resulting in four summaries per question (Expert, Sonnet, GPT-4o, Llama).
• Evaluation Protocol: The ten specialists acted as blinded evaluators, reviewing summaries for questions they did not author. Evaluation was conducted via a standardized rubric scoring 1–10 on four axes: Correctness (hallucinations, misinterpretations, fabricated citations), Completeness (missing concepts, references), Conciseness (redundancy, length), and Clinical Usefulness. Evaluators also ranked preferences, attempted to identify the human-written summary, and provided free-text feedback.

Key Results
Analysis of 200 critical assessments (1,800 annotated concepts) revealed the following:

• Quantitative Performance: Human experts achieved the highest scores across all metrics, followed by Sonnet, GPT-4o, and Llama 3.1.
  • Expert vs. Sonnet: No statistically significant difference was found between human experts and Sonnet in correctness, completeness, conciseness, or usefulness.
  • Expert vs. GPT-4o/Llama: Experts significantly outperformed GPT-4o and Llama 3.1 in most categories. Llama 3.1 scored significantly lower than experts in completeness, conciseness, and usefulness, with no significant difference only in correctness.
• Conciseness and Errors: Expert and Sonnet summaries were tied for the highest conciseness scores. Llama 3.1 was most frequently penalized for unnecessary and repeated information. GPT-4o and Llama 3.1 received the largest penalties for hallucinated or fabricated content.
• Human vs. AI Identification: Experts correctly identified human-written summaries 64% of the time (32/50). The most distinguishing features attributed to human writing were information focus, completeness, reference quality, and the inclusion of expert judgment.
• Preference: Human-written summaries were the most preferred (52% of top rankings), followed by Sonnet. Notably, in some instances, evaluators preferred an AI-generated summary (e.g., GPT-4o) for specific utility (e.g., providing dosing ranges) even when they correctly identified it as AI-generated.

Key Contributions

1. Direct Expert Comparison: This study provides a rare, direct comparison of RAG-enabled LLM outputs against expert-written literature summaries in a specialized medical field (Headache Medicine), moving beyond generic benchmarks to clinical utility.
2. Beyond Standard Metrics: The research demonstrates that standard quantitative metrics (correctness, completeness) are insufficient to capture expert preferences. Free-text analysis revealed that experts value structural flow (mimicking clinical reasoning), synthesis (inference over data quoting), reference authority (milestone trials, gold-standard guidelines), and clinical nuance (contraindications, ongoing trials).
3. Identification of AI Limitations: The study identified specific bottlenecks in current RAG systems, including the misinterpretation of literature, omission of key clinical details (e.g., specific dosages), and the failure to distinguish between a summary and a systematic review.
4. Framework Optimization: The paper details the iterative optimization of a RAG pipeline for medical summarization, emphasizing the necessity of human-in-the-loop prompt engineering to prioritize evidence levels and actionable clinical data.

Significance and Claims
The authors claim that while LLMs (particularly Sonnet) are approaching human performance in basic quantitative metrics, a "substantial and previously uncharacterized gap" remains in the qualitative depth, clinical relevance, and interpretive synthesis of the summaries. The study asserts that even when clinicians cannot reliably distinguish AI from human writing, they still prefer human-written summaries by a significant margin.

The paper concludes that future refinement of literature summarization pipelines must look beyond standard evaluation metrics. To achieve clinical utility, AI systems must incorporate features valued by experts: the ability to synthesize information rather than just retrieve it, the inclusion of high-authority references, the integration of clinical nuance and experience, and the ability to anticipate clinical questions. The findings suggest that while LLMs are powerful tools, they currently serve best as augmentations to, rather than replacements for, expert synthesis in high-stakes clinical decision-making.