Dissecting clinical reasoning failures in frontier artificial intelligence using 10,000 synthetic cases

This study utilizes 10,000 synthetic Multiple Sclerosis cases and automated expert evaluation to reveal that while frontier AI models can generate accurate differential diagnoses, they frequently commit catastrophic safety errors—such as recommending inappropriate thrombolysis or steroids—highlighting the critical need for massive-scale simulation to identify clinical blind spots before real-world deployment.

The Gist

Imagine you are hiring a new, super-smart medical intern to help diagnose patients. Before you let them treat anyone, you want to make sure they aren't just memorizing textbook answers but can actually think like a doctor in the messy, unpredictable real world.

This paper is about a team of researchers who decided to put the world's smartest AI "interns" (Large Language Models) through a massive, high-stakes driving test to see if they are ready to drive on the highway of healthcare.

Here is the breakdown of what they did and what they found, using some simple analogies:

1. The Problem: The "Driver's Ed" Trap

Usually, when we test AI doctors, we give them a small quiz with maybe 50 or 100 questions. It's like giving a new driver a test in an empty parking lot. They can memorize the answers, pass the test with flying colors, and then immediately crash when they hit a real street with traffic, rain, and unexpected obstacles.

The researchers knew that to truly test these AIs, they needed a massive simulation. They needed to throw thousands of weird, complex, and dangerous scenarios at the AI to see where it would break.

2. The Experiment: The "Synthetic Hospital"

Instead of using real patient records (which can be messy and private), the team built a virtual hospital using a computer program.

• The Patients: They generated 10,000 fake patients with Multiple Sclerosis (MS). These weren't just simple cases; the computer created patients with weird symptoms, confusing timelines, and hidden dangers (like active infections).
• The Ground Truth: The computer knew the exact right answer for every single fake patient (the "answer key").
• The Test: They asked four of the world's most advanced AI models to look at these fake patients and tell them:
  1. Where is the problem in the body? (Localization)
  2. What is the disease? (Diagnosis)
  3. What tests do we need? (Investigation)
  4. What medicine should we give? (Management)

3. The Results: The "Smart but Dangerous" Paradox

The results were shocking. It was like finding out a driver who can parallel park perfectly is also the one who accidentally drives into a lake because they forgot to check the water depth.

• The Good News (The Parking Lot): The AIs were great at the basics. They correctly identified that the patient likely had MS in over 90% of cases. They were smart enough to say, "Hey, this looks like a neurological issue."
• The Bad News (The Highway Crash): When it came to safety, the AIs failed miserably in ways that could kill a patient.
  • The Steroid Mistake: Sometimes, giving steroids (a common MS treatment) is dangerous if the patient has an active infection. The AIs often ignored the infection and said, "Give steroids now!"
  • The "Wrong Turn" Disaster: The most dangerous error was with thrombolysis (a powerful clot-busting drug used for strokes).
    • The Scenario: A patient comes in with symptoms that look like a stroke but are actually old MS symptoms (or just a random issue).
    • The AI Error: Two of the AI models (GPT-5.2 and GPT-5 mini) recommended giving this dangerous clot-busting drug to MS patients 10% of the time.
    • Why it matters: If you give a clot-buster to someone who doesn't have a clot, they could bleed to death. The AI didn't care that the symptoms were 14 days old or that the patient had MS; it just saw "neuro symptoms" and panicked, suggesting the wrong, dangerous treatment.

4. The "Blind Spots"

The researchers found that the AIs had specific "blind spots" that small tests would never catch.

• The Age Bias: The AI was more likely to suggest dangerous treatments for older patients.
• The Location Bias: If the problem was in a specific part of the brainstem, the AI was more likely to suggest the wrong drug.
• The "I Don't Know" Problem: When the AI didn't have enough information (like how long the symptoms had been there), it didn't say, "I need more info." Instead, it guessed and guessed dangerously.

5. The Conclusion: Why We Need a "10,000-Case" Test

The main takeaway is this: You cannot trust an AI doctor just because it gets an 'A' on a small quiz.

The researchers proved that by scaling up the test to 10,000 cases, they could find "rare, catastrophic failures" that happen only 1% or 2% of the time. In a small test of 50 cases, those failures would never show up. But in the real world, if an AI makes a fatal mistake 2% of the time, that's a disaster.

The Metaphor:
Think of the AI as a self-driving car.

• Old Testing: Driving the car around a quiet neighborhood for 10 minutes. It works great!
• This New Testing: Driving the car through a hurricane, on icy roads, with confused pedestrians, for 10,000 miles.
• The Finding: The car drives fine in the neighborhood, but in the storm, it occasionally tries to drive off a cliff.

The Bottom Line: Before we let AI treat real humans, we need to run these massive, automated "stress tests" to find the hidden cliffs and build guardrails. We need to know exactly when and why the AI fails so we can fix it before it hurts anyone.

Technical Summary

1. Problem Statement

Current evaluations of medical Large Language Models (LLMs) suffer from significant limitations that obscure safety risks before clinical deployment:

• Scale and Diversity: Existing benchmarks rely on small datasets (dozens of cases) that are easily "cheated" by models and fail to capture the complexity of real-world clinical "edge cases."
• Ground Truth Issues: Retrospective datasets lack definitive ground truth for unobserved clinical facts (e.g., a missed lesion on an un-ordered MRI) and carry a high risk of training-data contamination.
• Safety Blind Spots: High diagnostic accuracy on superficial tasks often masks catastrophic failures in management and safety (e.g., recommending contraindicated treatments). There is an urgent need for a method to rigorously stress-test AI reasoning at a scale impossible for human experts to review manually.

2. Methodology

The authors developed an automated, expert-validated simulation framework using Multiple Sclerosis (MS) as a complex clinical model due to its nuanced diagnostic criteria and management requirements.

• Synthetic Case Generation:

  • A neurologist-authored system programmatically generated 10,000 unique synthetic MS cases.
  • Cases were built on a "ground truth" foundation: specific neuroanatomical lesion locations were defined first, followed by the generation of corresponding symptoms, signs, and comorbidities.
  • The system introduced probabilistic noise and varied clinical contexts (e.g., incidental complaints, symptom duration >14 days, active infections) to test reasoning logic rather than pattern matching.
  • Cases were rendered as standardized clinical summary images.

• Model Evaluation:

  • Four frontier multimodal LLMs were tested: Gemini 3 Pro, Gemini 3 Flash, GPT-5.2, and GPT-5 mini.
  • Models were prompted to provide: anatomical localization, differential diagnosis, investigation plans, and management strategies (including "Start Now" vs. "Delay" directives).

• Automated Evaluation System:

  • A custom automated evaluator compared LLM outputs against the ground-truth labels.
  • It utilized a hybrid approach: deterministic term-matching (synonyms) and semantic comparison via a local LLM (GPT-OSS 20B) to assess clinical meaning.
  • The system was highly permissive, accepting clinically feasible alternatives rather than penalizing minor omissions, but strictly flagged safety violations (e.g., steroids during active infection, thrombolysis for non-stroke).

• Human Validation:

  • Two subspecialist neurologists blinded to ground truth validated the synthetic cases and the automated evaluator's accuracy on a subset of 70 cases.
  • This confirmed 100% case realism and 99.8% automated evaluation accuracy.

3. Key Contributions

• Massive-Scale Simulation: The study successfully scaled clinical reasoning evaluation to 10,000 cases, generating over 606,000 discrete clinical judgments. This scale is unattainable via manual human review.
• Decoupling Diagnosis from Safety: The framework demonstrates that high diagnostic accuracy does not correlate with safe management recommendations.
• Granular "Phenotyping" of AI Errors: By analyzing thousands of diverse scenarios, the study identified specific "blind spots" and systematic biases (e.g., age-related testing biases, lesion-location specific errors) that are invisible in small-scale benchmarks.
• Automated Expert-Level Interrogation: The system provides a reproducible, rapid method to uncover catastrophic failures (like inappropriate thrombolysis) before real-world deployment.

4. Key Results

A. Diagnostic Performance vs. Safety

• Diagnosis: All models successfully included MS in the differential diagnosis for >91% of cases. GPT-5.2 performed best (98.5%).
• Localization: Models performed well on optic nerve lesions (>87% exact match) but failed significantly on spinal cord localization (<10% exact match), often failing to lateralize or assign correct levels.
• Safety Failures (Management):
  • Steroids: Safe steroid recommendations were alarmingly low. Gemini 3 Flash recommended appropriate steroids in only 7.2% of cases; GPT-5 mini was better at 23.5%. Models frequently ignored contraindications like active infections or symptoms >14 days old.
  • Thrombolysis (Catastrophic Error): OpenAI models (GPT-5.2 and GPT-5 mini) inappropriately recommended acute intravenous thrombolysis for MS cases in 9.6% and 6.4% of cases, respectively. Gemini models were safer (<1%).
  • Contextual Blind Spots: Thrombolysis recommendations persisted even when symptoms were explicitly documented as >14 days old (2.9% of cases) or when no timing information was provided (10.1%).

B. Contextual Biases

• Age Bias: Models were more likely to recommend AQP4/MOG antibody testing for younger patients, with GPT-5 mini showing a strong bias (73.8% for <30s vs. 41.1% for >50s).
• Lesion Location Bias: GPT-5 mini failed to recommend AQP4/MOG testing for medullary lesions (34.1%) despite this being a hallmark site for Neuromyelitis Optica Spectrum Disorder (NMOSD), whereas other models did so in ~67% of cases.

C. Statistical Significance

• The large sample size allowed for the detection of rare, high-risk behaviors (e.g., thrombolysis errors) that would be statistically invisible in datasets of <1,000 cases.
• Cramer's V analysis confirmed that while some associations were statistically significant, the clinical magnitude of certain biases (like total lesion burden affecting thrombolysis decisions) was negligible, highlighting the need to distinguish mathematical artifacts from clinical reasoning flaws.

5. Significance and Implications

• Redefining AI Safety: The study proves that "frontier" status and high diagnostic scores do not guarantee clinical safety. Models can be "superhuman" at recall but "catastrophically flawed" in decision-making logic.
• Necessity of Stress-Testing: The authors argue that independent, rigorous stress-testing across thousands of diverse scenarios must become a prerequisite for clinical AI deployment.
• Guardrail Development: By characterizing the specific drivers of errors (e.g., ignoring symptom timing), developers can implement targeted safety guardrails rather than relying on broad, untargeted benchmarks.
• Methodological Shift: The paper advocates for a shift from retrospective, small-scale human review to prospective, massive-scale synthetic simulation with automated, expert-level evaluation to uncover hidden clinical blind spots.

Conclusion: The study concludes that without massive-scale simulation, dangerous AI failures will remain invisible until they harm patients. The proposed framework offers a scalable solution to map the entire landscape of clinical edge cases and ensure AI safety before real-world exposure.