🤖 AI

MedGemma vs GPT-4: Open-Source and Proprietary Zero-shot Medical Disease Classification from Images

This study demonstrates that the open-source, LoRA-fine-tuned MedGemma model significantly outperforms the proprietary GPT-4 in zero-shot medical disease classification from images, achieving higher accuracy and sensitivity while underscoring the critical importance of domain-specific fine-tuning for reliable clinical AI.

Original authors: Md. Sazzadul Islam Prottasha, Nabil Walid Rafi

Published 2026-04-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Md. Sazzadul Islam Prottasha, Nabil Walid Rafi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to diagnose a patient's illness using only a picture of their body (like an X-ray, MRI, or skin scan). You have two different "doctors" available to help you:

Dr. General (GPT-4): A brilliant, world-famous polymath who has read almost every book in the library. He knows a little bit about everything—history, cooking, math, and even some medicine. However, he has never gone to medical school, and he has never seen a specific patient's file before. He is guessing based on his general knowledge.
Dr. Specialist (MedGemma): A highly trained medical resident who has spent years studying specifically at a medical school. He has read millions of medical journals, studied thousands of patient records, and practiced on a specific set of images. He is an expert in his field.

This paper is a head-to-head competition between these two doctors to see who is better at diagnosing six different diseases (like skin cancer, pneumonia, and Alzheimer's) just by looking at medical images.

The Setup: The "Zero-Shot" Challenge

The researchers set up a tricky test.

Dr. General (GPT-4) was asked to look at the images and give an answer immediately, without any extra practice or studying for this specific test. This is called "zero-shot" learning. It's like asking a generalist to perform heart surgery just because he knows what a heart looks like from a textbook.
Dr. Specialist (MedGemma) was given a "crash course" (called fine-tuning). The researchers showed him thousands of examples of the specific diseases he needed to diagnose. They used a clever shortcut technique called LoRA (Low-Rank Adaptation), which is like giving the doctor a set of specialized highlighters and sticky notes to focus only on the most important details, rather than rewriting his entire brain.

The Results: Who Won?

The results were clear, much like a sports match where the home team (the specialist) dominates the visiting team (the generalist).

The Scoreboard: Dr. Specialist (MedGemma) got about 80% of the diagnoses right. Dr. General (GPT-4) only got about 70% right.
The Gap: In the world of medicine, a 10% difference is huge. It's the difference between a coach who wins the championship and one who barely makes the playoffs.

Why Did the Specialist Win?

The paper uses some great metaphors to explain why the generalist struggled:

The "Hallucination" Problem:
Dr. General (GPT-4) is so confident that sometimes he makes things up. If he doesn't know the answer, he might invent a plausible-sounding but wrong diagnosis. This is called "hallucinating." In a hospital, making up a diagnosis is dangerous. Dr. Specialist, having studied the specific patterns of disease, is much less likely to guess wildly.
The "Sequence Memory" Trap:
The paper suggests that Dr. General was sometimes just memorizing the order of the questions rather than truly understanding the picture. It's like a student who memorizes the answer key for a practice test but fails the real exam because they didn't actually learn the math. Dr. Specialist actually learned to see the disease.
The "Black Box" vs. The "Glass Box":
Traditional AI models are like black boxes—you put an image in, and a result comes out, but you don't know why. Dr. Specialist is more like a glass box; because he was trained on medical data, his reasoning is more grounded in clinical reality, making doctors trust him more.

The Verdict

The paper concludes that while general AI models (like GPT-4) are amazing tools for writing emails, coding, or chatting, they are not ready to replace doctors for serious medical diagnoses.

Think of it this way: You would ask a generalist to help you write a poem, but you would ask a specialist to perform surgery. To make AI safe for hospitals, it needs to be "specialized" (fine-tuned) just like a medical student needs to specialize in a specific field.

The Bottom Line:
If you want to diagnose a disease from an image, don't just ask the smartest person in the room who knows a little bit about everything. Ask the person who has studied that specific problem for years. In this race, the specialized, open-source model MedGemma beat the famous, proprietary GPT-4 by a significant margin, proving that in medicine, specialized training beats general knowledge.

1. Problem Statement

The study addresses the critical gap in applying Artificial Intelligence to medical diagnostics, specifically the limitations of general-purpose Large Language Models (LLMs) versus domain-specific multimodal models.

The Challenge: While general-purpose models like GPT-4 excel at broad linguistic tasks, they often lack the specialized clinical grounding required for accurate medical diagnosis. They are prone to "hallucinations" (fabricating medical facts), struggle with high-resolution medical imaging features, and fail to provide evidence-based reasoning necessary for clinical trust.
The Gap: There is a lack of rigorous, direct comparisons between specialized, open-source medical agents (like MedGemma) and proprietary generalist models (like GPT-4) across diverse, high-stakes medical imaging datasets.
Objective: To evaluate whether a medically fine-tuned, open-source model (MedGemma) outperforms a zero-shot, proprietary model (GPT-4) in classifying six distinct chronic diseases using medical imaging.

2. Methodology

The research employed a comparative experimental design involving two distinct AI architectures evaluated across six diverse medical datasets.

A. Datasets

Six datasets covering multiple clinical modalities (MRI, CT, X-ray, ECG, Dermatology) were utilized:

Skin Cancer (HAM10000): 10,015 images across 7 classes (e.g., Melanoma, Basal Cell Carcinoma).
Alzheimer's Disease (OASIS): 86,437 brain MRI slices classified by dementia severity (Non-demented to Moderate).
Breast Cancer (CBIS-DDSM): 3,568 mammography images (Normal, Benign, Malignant).
Cardiovascular Disease: 11,148 ECG images (Myocardial Infarction, Abnormal Heartbeat, etc.).
Pneumonia: 5,856 Chest X-ray images (Normal vs. Pneumonia).
Chronic Kidney Disease: 12,446 CT images (Normal, Cyst, Tumor, Stone).

B. Data Preprocessing

Normalization: All images were resized to 224x224 pixels and pixel intensities normalized to [0, 1].
Augmentation: Random rotations (±10°), horizontal flips, and intensity modifications (±20%) were applied to prevent overfitting.
Feature Extraction: A pre-trained Vision Transformer (ViT-B/16) was used to extract 768-dimensional feature embeddings from the images. These embeddings served as the input for the classification heads.

C. Model Architectures

MedGemma-4b-it (Experimental):
- A specialized, open-source multimodal model based on the Gemma architecture, fine-tuned for biomedical domains.
- Training Strategy: Fine-tuned using Low-Rank Adaptation (LoRA) to reduce trainable parameters by 10,000x, making training feasible on limited GPU resources (NVIDIA A100 cluster).
- Hyperparameters: AdamW optimizer, Learning Rate $2 \times 10^{-5}$ , Batch Size 32, 15 Epochs, Early Stopping.
- Loss Functions: Categorical cross-entropy for multi-class problems; Binary cross-entropy for binary problems.
GPT-4 (Baseline):
- A proprietary, general-purpose multimodal LLM.
- Evaluation Strategy: Zero-shot inference only. No fine-tuning was applied; the model relied solely on standardized natural language prompts to interpret the images.

D. Evaluation Metrics

Performance was assessed using Accuracy, Precision, Recall, F1-Score, and Confusion Matrices on a held-out test set (5-10% of data) for each disease category.

3. Key Contributions

Empirical Validation of Domain Specialization: The study provides quantitative evidence that domain-specific fine-tuning (MedGemma) significantly outperforms general-purpose zero-shot models (GPT-4) in medical imaging tasks.
Comprehensive Multi-Modal Benchmark: Unlike previous studies focusing on single modalities, this work evaluates performance across six distinct disease types and imaging technologies (MRI, CT, X-ray, ECG, Dermatology).
Analysis of "Hallucination" and Reasoning: The paper highlights that while GPT-4 can handle general queries, it lacks the deep visual feature recognition required for subtle medical patterns, leading to lower sensitivity in critical conditions like cancer and pneumonia.
Efficiency of LoRA: Demonstrates that open-source models can achieve high clinical performance with resource-efficient fine-tuning techniques (LoRA), offering a viable alternative to massive proprietary models.

4. Results

The study found a consistent performance gap favoring the fine-tuned MedGemma model across all six datasets.

Overall Accuracy:
- MedGemma: Mean test accuracy of 80.37%.
- GPT-4: Mean test accuracy of 69.58%.
- Gap: MedGemma outperformed GPT-4 by approximately 10–14% across all categories.
Specific Disease Performance Highlights:
- Pneumonia: MedGemma achieved 81.71% accuracy vs. GPT-4's 69.91%. MedGemma showed superior recall (83.6%), crucial for minimizing missed infections in emergency settings.
- Alzheimer's: MedGemma achieved 80.44% vs. GPT-4's 71.16%. MedGemma showed better differentiation between dementia severity stages.
- Breast Cancer: MedGemma achieved 81.11% vs. GPT-4's 70.45%. MedGemma demonstrated higher sensitivity in detecting malignant lesions (F1-score 81.6% vs. 71.5%).
- Cardiovascular (ECG): MedGemma achieved 79.34% vs. GPT-4's 67.65%. MedGemma showed significantly higher recall for Myocardial Infarction (79.9% vs. 69.4%).
- Chronic Kidney Disease: MedGemma achieved 80.57% vs. GPT-4's 68.76%.
Qualitative Analysis:
- Confusion matrices revealed that GPT-4 frequently misclassified subtle features (e.g., confusing benign vs. malignant tumors or different stages of dementia).
- GPT-4 appeared to rely on sequence memorization of prompts rather than deep visual feature extraction.
- MedGemma exhibited higher sensitivity in high-stakes classes (e.g., Melanoma, Malignant Breast Cancer), reducing false negatives.

5. Significance and Conclusion

Clinical Implications: The results suggest that for high-stakes clinical decision-making, general-purpose LLMs are insufficient without specialized training. The "black box" nature and lack of medical grounding in models like GPT-4 pose risks of misdiagnosis and hallucination.
Path Forward: The study advocates for domain-specific adaptation (fine-tuning) as an essential step for deploying AI in healthcare. Open-source models like MedGemma, when fine-tuned with techniques like LoRA, offer a cost-effective, transparent, and highly accurate alternative to proprietary black-box models.
Limitations & Future Work: The authors note that MedGemma's performance is constrained by the limited scale of the fine-tuning dataset and computational resources compared to the massive pre-training of GPT-4. Future work should focus on large-scale clinical validation, ethical integration into hospital workflows, and developing "hallucination-aware" training strategies to ensure safety and interpretability.

Final Verdict: The paper concludes that while GPT-4 is a powerful generalist, MedGemma is the superior tool for medical diagnostics, proving that specialized training is non-negotiable for reliable, evidence-based medical AI.