⚡ electrical engineering

OpenMedQ: Broad Open Pretraining for Medical Vision-Language Models

OpenMedQ is a state-of-the-art medical vision-language model pretrained on a broad, fully-open dataset of 3.35 million samples that outperforms significantly larger models like Med-PaLM M on key benchmarks while achieving superior performance in downstream medical classification tasks.

Original authors: Ibrahim Gulluk, Max Van Puyvelde, Olivier Gevaert

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ibrahim Gulluk, Max Van Puyvelde, Olivier Gevaert

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the complex world of medicine. Usually, to do this, you need two things: a "brain" that can read medical reports and a pair of "eyes" that can look at X-rays, microscopes, and pathology slides.

For a long time, the best robots (AI models) in this field have been like secretive geniuses. They are incredibly smart, but their creators keep their training books (data) and their brain weights hidden. You can't see how they learned, you can't reuse their knowledge, and you can't fix their mistakes. Other models are like specialized apprentices; they are great at one specific task (like reading X-rays) but haven't read enough books to understand the whole picture.

OpenMedQ is the paper's answer to this problem. It is a new, "open-source" medical robot that the authors built by gathering the largest collection of open, free medical textbooks and image sets ever assembled to train a model of this size.

Here is a breakdown of what they did and what they found, using simple analogies:

1. The "Library" (The Training Data)

Most medical AI models are trained on a small, narrow library of books. OpenMedQ, however, was trained on a massive, diverse library containing 14 different datasets with about 3.35 million examples.

Think of it like this:

Other models might only read books about heart X-rays.
OpenMedQ read books about heart X-rays, brain scans, microscope slides of cells, and even just text-based medical quizzes.
Crucially, every single book in this library was open and free for anyone to use. The authors didn't hide their sources.

2. The "Brain" and the "Eyes" (The Architecture)

The model is built like a standard modern AI (called LLaVA-style).

The Eyes (Vision Encoder): It uses a pre-trained "eye" that was already good at looking at medical images (from a model called BiomedCLIP).
The Brain (Language Model): It connects those eyes to a large language brain (LLaMA-7B) that was already good at medical text.
The Training: They taught the eyes and brain to talk to each other using a technique called "next-token prediction." Imagine showing the robot an image and a question, and asking it to guess the next word in the answer, over and over again, until it learns the pattern.

3. The "Test Drive" (The Results)

The authors put OpenMedQ through two main tests to see how well it learned.

Test A: The "General Knowledge" Quiz (Visual Question Answering)
They asked the robot medical questions based on images (e.g., "What does this X-ray show?").

The Result: OpenMedQ scored higher than some of the biggest, most expensive models out there.
The Analogy: Imagine OpenMedQ is a student with a 7B-parameter brain (a medium-sized brain). It took a test against a giant, 562B-parameter brain (a massive supercomputer brain). Despite being roughly 80 times smaller, OpenMedQ got a better score on one specific test (PathVQA) and matched the best scores on another (VQA-MED).
The Claim: This proves that having a broad, open library of training data is more important than just having a huge, secret brain.

Test B: The "Specialist" Exam (Image Classification)
They took just the "eyes" (the vision part) of OpenMedQ and tested it on 8 different medical image tasks it had never seen before (like identifying breast cancer in ultrasound or pneumonia in chest X-rays).

The Result: The OpenMedQ eyes performed better on average than the eyes of three other top-tier medical models (BiomedCLIP, PMC-CLIP, PubMedCLIP) and a model trained from scratch.
The Analogy: It's like taking a generalist doctor who has seen a little bit of everything and asking them to diagnose specific diseases. Because they saw such a wide variety of cases during their training, they were better at spotting patterns in new situations than doctors who only specialized in one narrow field.

4. The Catch (Limitations)

The authors are honest about where the robot still struggles.

Not Perfect Everywhere: While OpenMedQ was the best on average, it didn't win every single category. For example, on breast ultrasound images, another model was still slightly better.
Surface Level: The test scores (BLEU-1) measure how similar the robot's words are to human answers, not necessarily if the medical reasoning is 100% correct.
The "Big Brain" Still Wins Some: The massive, secret models (Med-PaLM M) still did better on some specific, difficult tests involving radiology and microscopy.

The Bottom Line

The main message of the paper is that diversity and openness are powerful tools. You don't necessarily need a secret, massive supercomputer to build a great medical AI. If you train a medium-sized model on the broadest, most open collection of medical data available, you can beat much larger, closed models.

The authors have made their code, their training recipes, and an interactive demo available to the public, inviting everyone to inspect, reuse, and improve upon their work.

Technical Summary: OpenMedQ: Broad Open Pretraining for Medical Vision-Language Models

Problem Statement

Despite the increasing capabilities of medical foundation models, a significant gap exists in the availability of fully open, broadly pretrained baselines. Most published medical Vision-Language Models (VLMs) rely on narrow pretraining sources and often withhold their weights, training data, or both. Existing contrastive encoders (e.g., BiomedCLIP, PMC-CLIP) are trained on single image-caption corpora, while generative VLMs (e.g., PMC-VQA, LLaVA-Med) utilize comparably narrow pretraining mixes. Although larger models like BiomedGPT and Med-PaLM M scale data and parameters, they do not release weights. Consequently, practitioners lack a transparent, reproducible baseline that is both broadly pretrained and fully open for inspection, reuse, and extension.

Methodology

The authors introduce OpenMedQ, a medical VLM designed to address these limitations through broad, open pretraining.

Architecture: OpenMedQ follows the LLaVA-style architecture, consisting of a vision encoder and a language model.
- Vision Encoder: A ViT-base-patch16-224 initialized from BiomedCLIP.
- Language Model: A LLaMA-7B model initialized from PMC-LLaMA.
- Integration: Image tokens are projected linearly and concatenated with text tokens, decoded left-to-right.
Pretraining Strategy: The model is trained using next-token prediction with a Low-Rank Adaptation (LoRA) rank of $r=8$ . During training, image and prefix tokens are masked to focus on next-token cross-entropy.
Training Configuration: Images are resized to 224×224. Training utilizes AdamW with a batch size of 64, a learning rate of $5\times10^{-5}$ , and runs for up to 15 epochs on a single NVIDIA A100.
Classification Transfer: To evaluate the quality of the learned vision features, the vision encoder ( $f_{vis}$ ) is detached and attached to a linear head ( $W \in \mathbb{R}^{2d \times m}$ ). Both are fine-tuned together on downstream datasets for 100 epochs. This setup ensures that performance differences are attributable to the pretraining rather than downstream architectural variations.

Key Contributions

Broadest Open Pretraining Mix: OpenMedQ is pretrained on the most extensive fully-open medical dataset mix to date, comprising 14 datasets and approximately 3.35 million samples. This mix spans:
- Pathology: PathVQA.
- Radiology: VQA-RAD, IU-XRAY, MIMIC-CXR, ROCO, OmniMedVQA.
- Microscopy: $\mu$ -Bench.
- Mixed Modalities: Slake, PMC-OA, PMC-VQA, VQA-MED.
- Text-Only Clinical QA: MedQA, MedMCQA, PubMedQA (approx. 410K samples) to preserve language capabilities.
Open Release: The authors commit to releasing model weights and dataset recipes upon acceptance, alongside a live interactive demo for qualitative inspection.
Reproducible Baseline: The paper establishes a standardized downstream recipe to fairly compare the impact of pretraining data diversity against strong baselines.

Results

The paper reports performance across two primary tasks: classification transfer and open-ended Visual Question Answering (VQA).

1. Classification Transfer

OpenMedQ's vision encoder was evaluated on 8 unseen medical classification benchmarks (CXR8, MedFMC subtasks, Breast-Ultrasound, CHAOYANG, CBIS-DDSM, Mendeley-CXray).

Performance: OpenMedQ achieved the highest average macro-F1 score of 0.757.
Comparison: This outperformed BiomedCLIP (0.745), PMC-CLIP (0.745), PubMedCLIP (0.746), and a from-scratch baseline (0.616).
Specific Wins: OpenMedQ achieved outright wins on MedFMC-chest and MedFMC-endo, tied PMC-CLIP on CXR8, and trailed the best encoder by at most 0.02 on four other benchmarks. The only significant gap was observed on Breast-Ultrasound (0.876 vs. 0.915).

2. Open-Ended VQA

PathVQA: OpenMedQ achieved a BLEU-1 score of 75.9, surpassing prefix tuning methods (70.3) and all three Med-PaLM M variants (up to 562B parameters, scoring up to 72.27), despite OpenMedQ using only 7B parameters.
VQA-MED: The model reached a BLEU-1 of 64.5, slightly exceeding the best reported score from the 2019 challenge (64.4).

Significance and Claims

The paper posits that breadth of open pretraining data is a critical competitive lever for medical VLMs. The authors claim that data diversity serves as a reproducible lever for performance, whereas proprietary scale is not.

Efficiency: OpenMedQ demonstrates that a 7B parameter model, when trained on a broad open mix, can set a new state-of-the-art on PathVQA against models up to 80 times larger (562B parameters).
Generalization: The vision encoder's superior performance on average classification transfer suggests that broad pretraining yields more robust feature representations than narrow, single-modality pretraining.
Limitations: The authors modestly acknowledge that their approach has limits. Larger proprietary variants (Med-PaLM M) still lead on specific benchmarks like VQA-RAD and Slake. Additionally, BLEU-1 captures only surface agreement, and narrow-modality encoders can still outperform OpenMedQ on specific tasks like Breast-Ultrasound.

The work concludes by emphasizing the value of open science, providing a transparent baseline that the community can inspect, reuse, and extend.