GPT-4o and DeepSeek-V3 responses to common migraine questions: a cross-sectional comparative study of accuracy, completeness, empathy, readability and safety

Technical Summary: Comparative Evaluation of GPT-4o and DeepSeek-V3 for Migraine Patient Education

Problem Statement
Patients increasingly rely on Large Language Models (LLMs) for health information, particularly for common conditions like migraine. However, migraine symptoms often overlap with secondary headache disorders (e.g., stroke, infection, intracranial pressure disorders), creating a risk where LLMs might provide factually correct but clinically incomplete or unsafe advice. While previous studies have assessed LLM accuracy, there is a gap in evaluating these models across a broader spectrum of clinical quality metrics, specifically completeness, empathy, readability, and safety regarding "red-flag" symptoms. This study addresses the need to compare the performance of two prominent models, GPT-4o and DeepSeek-V3, in the context of Turkish patient education for migraine.

Methodology
This cross-sectional comparative study utilized a dataset of 100 patient-facing questions derived from Google Trends, the r/migraine Reddit community, and major health institution resources (Mayo Clinic, Cleveland Clinic, NHS). The questions covered ten categories, including diagnosis, aura, triggers, treatment safety, and prognosis. Notably, 25 of these questions were prespecified as "red-flag" scenarios involving symptoms suggestive of secondary headaches (e.g., sudden worst-ever headache, neurological deficits, pregnancy-related headaches).

The study employed the following protocol:

• Data Collection: Questions were submitted unchanged to the public web interfaces of GPT-4o and DeepSeek-V3 in May 2026. No follow-up prompts or corrections were applied.
• Blinded Evaluation: Two neurologists, blinded to the model identity, independently rated the responses.
• Metrics:
  • Accuracy, Completeness, and Empathy: Scored on 1–5 scales. Accuracy measured agreement with current neurological practice; completeness assessed the inclusion of alternatives, uncertainty, and referral recommendations; empathy evaluated tone and acknowledgement of concern.
  • Safety: Scored on a 0–5 structured rubric focusing on red-flag recognition, urgent referral, and avoidance of harmful advice.
  • Readability: Calculated using the Ateşman formula (specific to Turkish language) to determine text difficulty.
• Statistical Analysis: Inter-rater agreement was measured using exact percentage and quadratic-weighted Cohen's kappa. Model comparisons used paired Wilcoxon signed-rank tests with Holm correction for multiple comparisons across the five primary outcomes.

Key Results
The study found high inter-rater agreement (quadratic-weighted kappa ranging from 0.810 to 0.920), validating the consistency of the expert ratings.

• Overall Performance: DeepSeek-V3 demonstrated statistically significant superiority over GPT-4o in completeness (3.98 vs. 3.46; adjusted p < 0.001) and readability (Ateşman score 74.75 vs. 60.70; adjusted p < 0.001). DeepSeek-V3 also showed higher accuracy (4.54 vs. 4.37; adjusted p = 0.026).
• Empathy and Safety: There was no significant difference in empathy between the models. While DeepSeek-V3 had a slightly higher mean safety score (4.67 vs. 4.54), this difference was not statistically significant after Holm correction (adjusted p = 0.075).
• Red-Flag Subgroup: In the 25 red-flag questions, both models achieved high safety scores with no significant differences in accuracy, safety, completeness, or empathy after correction. The only significant difference in this subgroup remained readability, favoring DeepSeek-V3.

Key Contributions

1. Multi-Dimensional Evaluation: Unlike prior studies focusing primarily on factual accuracy, this research provides a comparative analysis of LLMs across accuracy, completeness, empathy, safety, and readability simultaneously.
2. Red-Flag Specificity: The study explicitly isolates and evaluates responses to questions containing potential secondary headache warning signs, a critical safety metric often overlooked in general health queries.
3. Model Comparison: It introduces DeepSeek-V3 into the comparative landscape of medical LLMs, contrasting it with GPT-4o in a non-English (Turkish) context, thereby expanding the understanding of model performance across different languages and architectures.
4. Readability Quantification: The application of the Ateşman formula provides a quantitative measure of how accessible the medical information is to the average Turkish patient.

Significance and Claims
The authors conclude that both GPT-4o and DeepSeek-V3 generally provide accurate and safe information regarding migraine. However, DeepSeek-V3 offers more complete answers and is easier to read for Turkish speakers. The study emphasizes that while these models are useful for first-line education and health literacy, they should support, not replace, clinical assessment.

The paper explicitly states that neither model is uniformly superior in empathy or safety, and that even high safety scores in a static dataset do not guarantee safety in all dynamic clinical situations. The authors assert that LLMs should be used to explain general concepts and encourage professional consultation, particularly when red-flag symptoms are present, but must be subject to human supervision and critical review. The study acknowledges limitations, including the specific language (Turkish), the static nature of the question set, and the lack of formal validation for the study-specific rating rubrics.