Evaluating GPT-5 as a Multimodal Clinical Reasoner: A Landscape Commentary

This landscape commentary evaluates the GPT-5 family against GPT-4o, revealing substantial improvements in expert-level textual reasoning and multimodal synthesis that approach state-of-the-art performance in tasks like mammography, while highlighting that generalist models still lag behind specialized systems in perception-critical domains such as neuroradiology.

The Gist

Imagine you are trying to teach a brilliant, well-read student how to be a doctor. This student has read every medical textbook, every journal, and every case study ever written. But there's a catch: they've never actually looked at a real X-ray, a microscope slide, or a mammogram. They only know what the words say about those images.

This paper is a report card for the newest version of that student: GPT-5. The researchers wanted to see if this "super-student" has graduated from just memorizing facts to actually thinking like a doctor when looking at pictures and text together.

Here is the breakdown of their findings, using some everyday analogies:

1. The Test: A Medical "Obstacle Course"

The researchers didn't just ask GPT-5 simple questions. They put it through a grueling obstacle course with three different types of challenges:

• The Textbook Exam: Standard medical questions (like the USMLE board exams) to see if it knows the facts.
• The Detective Work: Complex cases where the student has to read a patient's story, look at a lab report, and then look at an image to solve a mystery.
• The Visual Eye-Test: Looking at specific medical images (brain MRIs, microscope slides of cells, and breast X-rays) and answering questions about what they see.

2. The Results: The "Smart Generalist" vs. The "Specialist"

🧠 The Textbook Genius (Text-Only Tasks)

The Verdict: GPT-5 is a phenomenal scholar.
When the test was just about reading and reasoning with words, GPT-5 crushed it. It scored over 95% on medical board exams.

• The Analogy: If the previous model (GPT-4o) was a smart college graduate, GPT-5 is a tenured professor who can explain complex medical concepts without stumbling. It didn't just memorize the answers; it figured out the logic behind them.

🔍 The Detective (Combining Text + Images)

The Verdict: GPT-5 is getting much better at connecting the dots.
When the test required looking at a picture and reading a story to make a diagnosis, GPT-5 showed a massive leap forward.

• The Analogy: Imagine a detective who used to only read police reports. Now, GPT-5 can look at the crime scene photo and read the witness statement to figure out what happened. In one test, it correctly identified a rare, life-threatening condition (a torn esophagus) by combining a CT scan image with the patient's vomiting history. It acted like a seasoned clinician, not just a search engine.

👁️ The Visual Specialist (Looking at Images Alone)

The Verdict: GPT-5 is good, but not a specialist yet.
This is where the "super-student" hit a wall.

• Brain Scans (Neuroradiology): GPT-5 got about 44% of the answers right. It's better than a layperson, but a real brain surgeon would get 90%+. It's like a general mechanic trying to fix a Formula 1 engine; they know how engines work, but they miss the tiny, specific details.
• Microscopes (Pathology): It did okay, but sometimes it got confused by the tiny details of blood cells or tissue samples.
• Mammograms (Breast X-rays): This was the hardest test. GPT-5 got about 50-60% right. Specialized AI systems built just for breast cancer get over 80-90%.
• The Analogy: Think of GPT-5 as a general practitioner (a family doctor). They are great at handling most things and can spot when something is wrong. But if you need a micro-surgeon to remove a tiny tumor or a radiologist to spot a tiny speck of calcium in a breast, you still need a human expert or a machine built specifically for that one job. GPT-5 is too "general" to be the best at the most "specialized" visual tasks.

3. The "Mini" Surprise

Interestingly, the smaller versions of GPT-5 (called "Mini" and "Nano") sometimes did better than the big "GPT-5" on specific image tests.

• The Analogy: It's like a giant, cautious elephant (GPT-5) vs. a nimble mouse (GPT-5 Mini). The elephant is incredibly smart and powerful, but sometimes it overthinks or is too careful when looking at small details. The mouse is faster and sometimes just "guesses" the right answer more often on small, specific tasks.

4. The Bottom Line: A Powerful Assistant, Not a Replacement

The paper concludes that GPT-5 is a huge step forward. It is the first time a general AI has shown it can truly "think" across text and images in a way that mimics a doctor's brain.

• What it can do: It's an amazing assistant. It can read a patient's history, look at a scan, and say, "Hey, this looks suspicious, here is why, and here is what we should check next." It helps doctors think faster and more holistically.
• What it can't do yet: It is not ready to replace a specialist. If you need a machine to look at a mammogram and decide if it's cancer with 99% certainty, you still need a tool built specifically for that, not a general-purpose AI.

In short: GPT-5 is like a brilliant medical student who is ready to start their residency. They are smart, they reason well, and they are improving fast. But they still need a supervising doctor (or a specialized tool) to handle the most critical, high-stakes visual decisions.

Technical Summary

1. Problem Statement

The transition from narrow, task-specific AI to general-purpose foundation models (FMs) raises critical questions regarding their ability to support integrated clinical reasoning. Real-world medicine requires synthesizing heterogeneous data sources—ambiguous patient narratives, laboratory results, and multimodal imaging (MRI, pathology slides, mammograms)—into cohesive diagnostic conclusions. While previous Large Language Models (LLMs) have shown promise in text-based recall, it remains unclear whether a single, generalist instruction-following model can reliably perform multimodal clinical reasoning without extensive domain-specific fine-tuning. This paper addresses the gap in understanding how next-generation models (specifically the GPT-5 family) handle the synthesis of text and complex medical imagery compared to their predecessors and specialized systems.

2. Methodology

The authors conducted a controlled, cross-sectional evaluation of the GPT-5 family (GPT-5, GPT-5 Mini, GPT-5 Nano) against its predecessor, GPT-4o.

• Protocol: All models were evaluated under a strict zero-shot regime with no fine-tuning and no in-context examples.
• Prompting Strategy: A standardized Chain-of-Thought (CoT) protocol was used.
  • Step 1: The model is anchored as a "medical assistant" and asked to generate a free-form rationale ("Let's think step by step").
  • Step 2: A forced convergence prompt ("Therefore, among A through {END LETTER}, the answer is") compels the model to select a single discrete option.
  • This design isolates model-level reasoning capabilities from prompt engineering artifacts.
• Datasets & Domains: The evaluation spanned 14 datasets across three clinically distinct imaging modalities and text-based reasoning:
  1. Medical Education & Text Reasoning: USMLE (Steps 1, 2, 3), MedQA, MMLU (Medical subdomains), and MedXpertQA (Text and Multimodal subsets).
  2. Neuroradiology: BraTS cohorts (Glioblastoma, Meningioma, Metastases) using multiparametric MRI. Images were processed into triplanar mosaics (axial, sagittal, coronal) with extracted clinical text.
  3. Digital Pathology & Hematology: PathVQA (Whole Slide Images, diagrams, gross specimens), Blood Cell VQA, and BreaKHis (breast histopathology).
  4. Mammography: Four datasets (EMBED, InBreast, CMMD, CBIS-DDSM) covering lesion characterization (mass, calcification, distortion), BI-RADS density, and malignancy classification.

3. Key Contributions

• First Controlled Evaluation of GPT-5 in Medicine: This is the first study to benchmark the GPT-5 family against GPT-4o across a diverse spectrum of clinically grounded multimodal tasks.
• Standardized Zero-Shot Framework: The authors established a rigorous, reproducible pipeline using CoT prompting to evaluate reasoning depth rather than just pattern matching.
• Granular Performance Analysis: The study dissects performance not just by accuracy, but by specific clinical tasks (e.g., lesion characterization vs. general diagnosis) and modalities, highlighting where generalist models succeed or fail compared to specialized systems.
• Identification of Reasoning vs. Perception Gaps: The paper distinguishes between the model's ability to reason through complex clinical narratives (where it excels) and its ability to perceive fine-grained visual details in specialized imaging (where it lags behind domain-specific models).

4. Key Results

A. Text-Based Reasoning & Medical Education

• USMLE: GPT-5 achieved a macro-average accuracy of 95.22% (a +2.88% improvement over GPT-4o), with the largest gains in Step 2 CK (+4.17%), indicating superior clinical decision-making capabilities.
• MedXpertQA (Expert Reasoning): GPT-5 demonstrated massive leaps in expert-level reasoning.
  • Text-only: +26.33% improvement in reasoning accuracy over GPT-4o.
  • Multimodal: +29.26% improvement in reasoning accuracy.
  • This suggests GPT-5 has significantly enhanced capabilities in multi-step inference and synthesizing complex medical narratives.

B. Multimodal Visual Question Answering (VQA)

• Neuroradiology (BraTS): Performance remained moderate. GPT-5 achieved a macro-average of 43.71%, slightly trailing GPT-5 Mini (44.19%) and GPT-4o (41.49) in some sub-cohorts. This indicates that generalist models still struggle with the specific visual features required for brain tumor differentiation.
• Digital Pathology:
  • PathVQA: GPT-5 achieved the best weighted accuracy (70.9%), showing improved robustness across different image sources (WSI, diagrams, gross specimens).
  • Blood Cell VQA: GPT-4o outperformed GPT-5 (80.0% vs. 75.2%), suggesting that specific cytological cues (nuclear lobulation, granularity) may still favor the predecessor or require specialized tuning.
• Mammography:
  • GPT-5 showed substantial improvements over GPT-4o in fine-grained tasks (e.g., +32.5% on BI-RADS density in EMBED).
  • However, absolute performance remained moderate (52–64% accuracy).
  • Comparison to Specialized Models: GPT-5 significantly underperformed compared to state-of-the-art domain-specific models. For example, on malignancy detection in EMBED, GPT-5 scored 52.8% compared to 82.3% for Mammo-CLIP.

C. Anomalies

• In several datasets (VQA-RAD, BraTS, EMBED, InBreast, CMMD), GPT-5 Mini slightly outperformed the full GPT-5 model (by 0.5% to 10%). The authors hypothesize this may be due to "reasoning calibration," where larger models adopt a more conservative approach on smaller, specific datasets.

5. Significance and Conclusion

• Advancement in Integrated Reasoning: GPT-5 represents a meaningful step toward holistic clinical reasoning. It effectively leverages enhanced textual reasoning to ground uncertain clinical narratives in imaging evidence, mimicking the clinician's cognitive process of biasing subjective information with objective findings.
• Limitations in Specialized Perception: Despite gains, generalist models are not yet substitutes for purpose-built systems in highly specialized, perception-critical tasks (e.g., mammography, brain tumor segmentation). The gap between GPT-5 (~50-60%) and specialized models (>80%) in these domains highlights the necessity of domain adaptation.
• Clinical Deployment Readiness: The study concludes that while GPT-5 is a powerful adjunct for clinical workflows, it requires:
  1. Domain Adaptation: Fine-tuning on specific medical imaging modalities.
  2. Rigorous Validation: Ensuring performance holds across diverse populations and datasets.
  3. Transparency: Mechanisms to quantify uncertainty and ensure reasoning fidelity (avoiding "hallucinated" reasoning patterns).

In summary, GPT-5 establishes a robust foundation for multimodal clinical inference but currently functions best as a high-level reasoning engine that must be paired with specialized, perception-optimized AI for critical diagnostic tasks.