Can Generalist Vision Language Models (VLMs) Rival Specialist Medical VLMs? Benchmarking and Strategic Insights

1. Problem Statement

The rapid development of Vision-Language Models (VLMs) in healthcare has led to two distinct approaches:

• Specialist Medical VLMs: Models pre-trained or fine-tuned extensively on medical corpora (e.g., BioMedCLIP, LLaVA-Med, Med-Gemma). While they show strong performance on specific in-distribution (ID) tasks, they require substantial computational resources, curated data, and are often expensive to develop and deploy.
• Generalist VLMs: Models trained on massive, diverse, non-medical web-scale datasets (e.g., CLIP, LLaVA, Gemma). They are widely available and easily adaptable via lightweight fine-tuning but lack explicit medical domain knowledge.

The Core Question: Is the expensive, resource-intensive development of specialist medical VLMs necessary, or can efficiently adapted generalist VLMs achieve comparable or superior performance on critical medical imaging tasks?

Existing benchmarks fail to answer this because they:

1. Overlook the effects of adaptation (fine-tuning), focusing only on off-the-shelf (OTS) performance.
2. Lack systematic, paired comparisons between generalist and specialist models within the same architectural family.
3. Do not adequately evaluate generalization to out-of-distribution (OOD) modalities.

2. Methodology: MedVLMBench

The authors introduce MedVLMBench, a unified benchmark designed to systematically compare generalist and specialist VLMs across three research questions (RQs).

Datasets and Scope

• Scope: 10 diverse medical datasets covering four imaging modalities: Radiology, Pathology, Dermatology, and Ophthalmology.
• Tasks: 144 diagnostic classification tasks and 72 Visual Question Answering (VQA) tasks.
• In-Distribution (ID) vs. Out-of-Distribution (OOD):
  • ID: Tasks where the primary modality was likely represented during the model's pretraining.
  • OOD: Tasks involving unseen modalities or domains relative to the model's pretraining.

Models Evaluated

The study assesses 18 VLMs paired across two paradigms:

1. Contrastive Models: Paired Generalist (CLIP, BLIP2, SigLIP) vs. Specialist (MedCLIP, PubMedCLIP, PLIP, BioMedCLIP, MedSigLIP).
2. Generative Models: Paired Generalist (LLaVA-1.5, Gemma3, Qwen-VL, InternVL3) vs. Specialist (LLaVA-Med, MedGemma, Lingshu).

• Note: Proprietary models (OpenAI o3, Gemini 2.5 Pro) were included for OTS comparison but excluded from fine-tuning experiments.

Evaluation Protocols

The study evaluates models under three conditions to address the RQs:

1. Off-the-Shelf (OTS): Zero-shot evaluation without adaptation.
2. Lightweight Adaptation (PEFT):
   • Contrastive: Linear Probing (LP) or LoRA.
   • Generative: LoRA applied to the bridge module and language decoder.
3. Generalization: Evaluating adapted models on OOD datasets.

Metrics

• Diagnosis: Area Under the Receiver Operating Characteristic Curve (AUROC).
• VQA: Token-level matching (Exact Match, F1, BLEU-1, ROUGE-L) and a GPT-based Semantic Score to evaluate semantic correctness and reduce bias from stylistic variations.

3. Key Contributions

1. First Systematic Paired Benchmark: MedVLMBench provides the first head-to-head comparison of generalist and specialist VLMs within the same model families across 10 datasets.
2. Adaptation-Centric Analysis: It shifts the focus from static OTS performance to the dynamic performance gains achievable through parameter-efficient fine-tuning (PEFT).
3. Strategic Insights on Generalization: It quantifies the trade-offs between domain-specific pretraining and the flexibility of generalist foundations, particularly regarding OOD generalization.

4. Key Results

RQ1: The OTS Gap (Specialist Advantage)

• Finding: Specialist medical VLMs consistently outperform generalist VLMs in the OTS setting on ID tasks.
• Evidence: For example, MedCLIP achieved 90.60% AUROC on CheXpert, significantly outperforming the generalist CLIP (40.95%). Similarly, MedGemma outperformed Gemma3 on VQA-RAD by nearly 12 GPT Score points.
• Implication: Domain-specific pretraining provides a strong initial advantage for tasks closely matching the training data.

RQ2: The Effect of Lightweight Fine-Tuning

• Finding: Lightweight fine-tuning (e.g., Linear Probing, LoRA) enables generalist VLMs to match or exceed the performance of specialist models on ID tasks.
• Evidence:
  • On the Camelyon dataset, the generalist BLIP2 jumped from a near-random OTS baseline to >98% AUROC after tuning, surpassing the specialist PLIP.
  • In VQA, tuned LLaVA-1.5 surpassed LLaVA-Med on PathVQA.
  • Across most datasets, tuned generalists outperformed both the OTS and tuned versions of their specialist counterparts.
• Implication: Broad visual-textual pretraining in generalists contains sufficient priors that can be efficiently specialized, potentially obviating the need for expensive domain-specific pretraining.

RQ3: Generalization to OOD Tasks

• Finding: Fine-tuned generalist VLMs demonstrate superior generalization on OOD datasets compared to specialist models.
• Evidence:
  • On dermatology (HAM) and multimodal VQA (SLAKE), tuned generalists (e.g., BLIP2, Qwen2.5-VL) significantly outperformed tuned specialists.
  • Specialist models often exhibited performance degradation or over-specialization when adapted to new modalities, whereas generalists retained flexible representations that transferred better.
• Implication: Specialist models may overfit to modality-specific features, while generalist models maintain a broader abstraction capability that aids cross-modal transfer.

5. Significance and Conclusion

• Cost-Efficiency: The study suggests that the "specialist" pipeline is not strictly necessary for state-of-the-art performance. A cost-effective strategy involves taking a strong generalist foundation and applying lightweight PEFT, which yields competitive or superior results with lower computational and data curation costs.
• Clinical Deployment: For clinical AI, where tasks vary and data modalities change, generalist models offer greater flexibility and scalability. They are more robust to OOD shifts, a critical requirement for real-world deployment where models encounter unseen imaging protocols or rare diseases.
• Future Direction: The paper advocates for a shift in medical AI development from building monolithic, domain-specific models to leveraging adaptable generalist foundations, provided that rigorous evaluation protocols (like MedVLMBench) are used to ensure safety and efficacy.

Limitations: The study defines ID/OOD based on modality, not finer distribution shifts (e.g., institution-specific artifacts). Additionally, the reliance on LLM-based scoring (GPT Score) for VQA, while standard, has inherent limitations. The results are specific to the models and datasets tested; different training curricula could alter relative performance.