Medical errors in large language models revealed using 1,000 synthetic clinical transcripts

This study reveals that despite achieving high diagnostic accuracy on full histories, large language models exhibit critical safety failures—including discouraging essential investigations and inappropriately downgrading triage for life-threatening conditions, with significant gender disparities—when evaluated against 1,000 synthetic clinical transcripts that simulate real-world medical complexity.

The Gist

Imagine you are training a new chef to cook a complex meal. The traditional way to test them is to give them a recipe card with perfect, step-by-step instructions for a classic dish like "Spaghetti Carbonara." If they make it perfectly, you assume they are a great chef.

But in the real world, customers don't give you perfect recipe cards. They say things like, "I'm hungry, my stomach hurts, and I think I ate something weird yesterday, but I'm not sure what." They might be distracted, speak with a heavy accent, or forget to mention they have a severe allergy.

This paper is about a team of researchers who decided to stop testing medical AI (specifically Large Language Models or LLMs) with perfect "recipe cards." Instead, they built a massive, high-tech simulator to create 1,000 different, messy, real-world conversations between a doctor and a patient.

Here is the breakdown of what they found, using simple analogies:

1. The "Perfect vs. Messy" Test

The researchers created 1,000 fake patients with headaches. Some had simple tension headaches; others had life-threatening brain bleeds.

• The Perfect Scenario: They gave the AI the full story (100% of the information).
• The Messy Scenario: They gave the AI only a tiny slice of the story (20% of the information), mimicking a patient who is confused, in pain, or can't remember details.

The Result:
When the AI got the full story, it was a genius. It diagnosed the problem correctly 97.5% of the time. It was like a chef who can cook a perfect meal when handed a perfect recipe.

But here is the scary part: When the story was incomplete, the AI didn't say, "I need more info to be safe." Instead, it guessed confidently and dangerously.

2. The "Overconfident Guess" Problem

In real medicine, if a doctor doesn't have enough information, they act like a cautious detective. They say, "I can't rule out a brain bleed yet, so let's run a test."

The AI, however, acted like a reckless gambler.

• The Lumbar Puncture Failure: For patients with a specific type of brain bleed (subarachnoid haemorrhage), the standard test is a spinal tap (lumbar puncture). When the AI didn't have the full timeline of the patient's pain, it didn't say, "I don't know." It confidently said, "No, don't do the spinal tap."
• The Triage Trap: For life-threatening emergencies, the AI often told the patient to "go home and rest" or "wait a few months" instead of telling them to go to the Emergency Room immediately. In fact, when information was missing, the AI told nearly 55% of these emergency patients to just manage it themselves.

3. The "Gender Bias" Glitch

The study found that the AI wasn't just making random mistakes; it had a bias.

• The Metaphor: Imagine a security guard who is extra careful with men but assumes women are just "overreacting."
• The Reality: The AI was significantly more likely to tell female patients (especially those aged 30–50) to go home and self-manage, even when they had symptoms of a life-threatening condition. It was 3 times more likely to make this dangerous mistake with women than with men.

4. The "Mini" vs. "Pro" Model

The researchers tested two versions of the AI: a "Pro" version (GPT-5.2) and a "Mini" version (GPT-5-mini).

• The Analogy: Think of the "Pro" version as a senior specialist doctor and the "Mini" version as a very fast, cheap intern.
• The Finding: The "Mini" version was much worse. It recommended dangerous painkillers (like codeine) more often, missed diagnoses more frequently, and was more likely to send emergency patients home.
• Why it matters: Many public health apps and search engines are using these cheaper, "Mini" models. This study suggests that using the cheaper model for health advice is like hiring the intern to perform surgery.

5. The Core Flaw: "Absence of Evidence"

The biggest lesson from this paper is how AI handles missing information.

• Human Logic: If I don't see a fire, but I smell smoke and hear a siren, I assume there might be a fire and I call the fire department. (We assume the worst to stay safe).
• AI Logic: If I don't see a fire in the data, I assume there is no fire. (It assumes that if a symptom isn't explicitly written down, the disease doesn't exist).

This is called "confusing absence of evidence with evidence of absence." Because the AI is a probability machine, it looks at the missing data and thinks, "Statistically, this is probably just a normal headache," rather than "This is a mystery, and mysteries could be deadly."

The Bottom Line

This paper is a wake-up call. It shows that while AI is great at reciting medical textbooks, it is currently terrible at handling the messy, incomplete reality of real human patients.

If we let these AI tools into hospitals or health apps without fixing this "reckless guessing" behavior, they could send people with brain bleeds home to die, or tell women their pain is "just in their head" when it's actually a medical emergency.

The researchers are calling for a new kind of "stress test" for medical AI—one that doesn't just ask, "Can you solve a puzzle?" but rather, "Can you stay safe when you don't have all the pieces?"

Technical Summary

1. Problem Statement

Current clinical evaluations of Large Language Models (LLMs) in healthcare rely heavily on small, curated datasets of "textbook" case vignettes. These summaries fail to capture the complexity of real-world medical encounters, which are characterized by:

• Fragmented and incomplete histories: Patients often provide partial information.
• Diverse communication styles: Variations in literacy, verbosity, and framing.
• Atypical presentations: Rare or "edge" cases that deviate from standard patterns.

The authors argue that high accuracy on textbook cases does not guarantee safety in real-world scenarios, particularly for life-threatening emergencies. There is a critical lack of rigorous, high-throughput evaluation frameworks capable of stress-testing LLMs against the "messy" reality of clinical practice, specifically regarding risk calibration and the handling of incomplete data.

2. Methodology

The study introduces a high-throughput patient simulation engine designed to generate naturalistic, controlled clinical data at scale.

• Simulation Engine Architecture:

  • Ground Truth: The engine generated 1,000 unique synthetic doctor-patient transcripts based on 33 distinct headache and facial pain diagnoses (ranging from benign tension headaches to life-threatening subarachnoid haemorrhage and meningitis).
  • Diagnostic Logic: Cases were grounded in the International Classification of Headache Disorders, 3rd edition (ICHD-3). A "ground-truth" diagnosis was established first, and phenotype facts were sampled to strictly satisfy ICHD-3 criteria.
  • Patient Personas: To simulate real-world communication diversity, the engine varied seven communication traits for each patient: Verbosity, Health Literacy, English Proficiency, Memory Reliability, Focus, Disclosure, and Framing (e.g., catastrophizing vs. stoic).
  • Controlled Incompleteness: A key innovation was the systematic variation of information completeness. Transcripts were generated at five sampling rates (20%, 40%, 60%, 80%, 100%), simulating scenarios where 80% of clinical data was missing.
  • Generation Process: A local LLM (GPT-OSS 20B) acted as the patient, responding to a standardized, pre-defined doctor script while adhering to the specific phenotype facts and persona constraints.

• Evaluation Protocol:

  • Models Tested: Two frontier models were evaluated: GPT-5.2 (state-of-the-art reasoning) and GPT-5-mini (reportedly used in public-facing health tools).
  • Tasks: For each transcript, models were tasked with:
    1. Generating a prioritized differential diagnosis.
    2. Recommending diagnostic investigations (CT, MRI, Lumbar Puncture).
    3. Recommending pharmacological interventions (Indomethacin, Codeine).
    4. Determining triage urgency (Immediate, Urgent, Routine, Self-management).
  • Scale: The study analyzed 14,000 independent clinical decisions derived from nearly 2.9 million words of dialogue.

3. Key Contributions

• Novel Evaluation Framework: Shifts medical AI evaluation from static "snapshot" testing to a dynamic, large-scale stress-test using synthetic, naturalistic transcripts with known ground truths.
• Quantification of Risk Calibration: Demonstrates that high diagnostic accuracy does not equate to safe clinical reasoning, specifically when data is incomplete.
• Demographic Bias Detection: Uncovered significant gender-based disparities in triage recommendations that were invisible in standard evaluations.
• Methodological Transparency: Provides a reproducible logic for generating synthetic clinical data without releasing the raw dataset (to prevent future model contamination), allowing for continuous, automated post-deployment monitoring.

4. Key Results

A. Diagnostic Accuracy vs. Information Completeness

• High Accuracy with Full Data: GPT-5.2 achieved 97.5% accuracy with complete (100%) histories.
• Degradation with Missing Data: As information decreased, accuracy for rare/emergency conditions (e.g., subarachnoid haemorrhage, meningitis) dropped significantly (to <50% at 20% sampling).
• Failure to Expand Differentials: Contrary to clinical intuition, models did not broaden their differential diagnoses when information was sparse. GPT-5.2 maintained a constant number of diagnoses, while GPT-5-mini actually narrowed its list as data became incomplete.

B. Investigation Recommendations (Overconfidence)

• Refusal to Defer: Safe clinical practice involves deferring decisions when data is insufficient. Both models failed to do this.
  • In Subarachnoid Haemorrhage (SAH) cases where the onset time was unknown (making a Lumbar Puncture decision impossible), models confidently recommended against the procedure in 100% of cases with low information (≤40% sampling), even when SAH was listed in the differential.
  • Models frequently recommended against necessary Lumbar Punctures for meningitis and idiopathic intracranial hypertension.

C. Pharmacological Safety

• Indomethacin: GPT-5-mini dangerously recommended indomethacin for SAH cases (a contraindication) in 33.3% of low-information scenarios.
• Codeine (Opiates): GPT-5-mini inappropriately recommended codeine in 16.1% of all cases, rising to 21.7% in low-information scenarios. It failed to adjust for age, recommending codeine equally for patients over 70 (high risk) and under 30.

D. Triage and Demographic Bias

• Downgrading Urgency: In life-threatening emergencies (SAH, meningitis), incomplete information caused models to downgrade triage to "self-management" or "routine follow-up."
  • GPT-5-mini: Unsafe triage rose from 9.3% (100% data) to 54.8% (20% data).
  • GPT-5.2: Unsafe triage rose from 7.0% to 42.9%.
• Gender Bias: GPT-5.2 was significantly less safe for female patients. Females were 3.2 times more likely (OR 3.2, 95% CI 1.4–7.1) to receive an unsafe triage recommendation compared to males, particularly in the 30–50 age group.

5. Significance and Implications

• The "Absence of Evidence" Fallacy: The study reveals a fundamental flaw in LLM reasoning: they conflate an absence of evidence (missing symptoms in the transcript) with an absence of disease. Human clinicians react to missing data by lowering thresholds for investigation; LLMs react by becoming overconfident in benign diagnoses.
• Model Selection Matters: The stark performance gap between GPT-5.2 and GPT-5-mini suggests that "mini" or computationally efficient models are dangerously unsuited for clinical decision support, despite their potential integration into public health tools.
• Need for Algorithmic Guardrails: Current probabilistic models lack the "deductive caution" required in medicine. Future AI integration requires explicit guardrails that force models to prioritize ruling out "must-not-miss" pathologies over statistical likelihood.
• Future of AI Evaluation: The authors argue that traditional accuracy metrics are insufficient. Safety must be validated through large-scale stress-testing across diverse, incomplete, and demographic-varied scenarios before deployment.

Conclusion: The paper concludes that while frontier LLMs show high diagnostic potential, they currently possess lethal reasoning failures regarding risk calibration and incomplete data. Without rigorous, high-throughput stress-testing and algorithmic recalibration, deploying these models in clinical settings poses significant risks of misdiagnosis and delayed care, particularly for women and patients with atypical or incomplete histories.