💻 computer science

MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Images

MedMO is a medical multimodal foundation model that leverages a specialized multi-stage training recipe—including cross-modal pretraining, multi-task instruction tuning, and reinforcement learning with verifiable rewards—to achieve state-of-the-art performance in medical image understanding, reasoning, and spatial grounding across diverse clinical domains.

Original authors: Ankan Deria, Komal Kumar, Adinath Madhavrao Dukre, Eran Segal, Salman Khan, Imran Razzak

Published 2026-03-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ankan Deria, Komal Kumar, Adinath Madhavrao Dukre, Eran Segal, Salman Khan, Imran Razzak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant medical student who has read every textbook in the library but has never actually seen a patient or looked at an X-ray. They know the words "pneumonia" and "fracture," but if you show them a blurry picture of a lung, they might guess wildly or make things up because they lack real-world experience.

This is the problem with current AI models in medicine. They are smart, but they are often "hallucinating" (making things up) because they haven't been trained specifically on the messy, complex reality of medical images.

Enter MedMO (Medical Multimodal Open). Think of MedMO not just as a student, but as a super-intern who has undergone a rigorous, four-stage "boot camp" to become a master diagnostician.

Here is how the paper explains MedMO's journey, using simple analogies:

1. The Problem: The "Generalist" vs. The "Specialist"

Current AI models are like general practitioners who have read a little bit about everything. They are great at chatting about movies or writing poems, but when you show them a microscopic image of bacteria or a CT scan of a brain, they often get lost. They might say, "That looks like a tumor," when it's actually just a shadow, or they might miss a fracture entirely.

MedMO was built to fix this. It is a specialist trained exclusively on medical data, designed to not just talk about medicine, but to see and understand it.

2. The Four-Stage Training Boot Camp

The researchers didn't just dump data on the model; they taught it in four distinct phases, like leveling up in a video game:

Stage 1: The "Library" Phase (General Medical SFT)
- The Analogy: Imagine the intern reading 18.5 million medical textbooks and case studies.
- What happened: The model learned the basics. It connected images (like X-rays) with text (like doctor's notes). It learned that a dark spot on a lung image usually means "pneumonia," not "a cloud." This gave it a massive foundation of medical knowledge.
Stage 2: The "High-Res Microscope" Phase (Grounding)
- The Analogy: Now, the intern is handed a high-powered microscope and a ruler. They aren't just reading; they are pointing.
- What happened: This is the most unique part of MedMO. The model was trained to point at things. If you ask, "Where is the broken bone?", MedMO doesn't just say "It's in the arm." It draws a box around the exact spot on the image. This is called "grounding." It forces the AI to be precise, not just vague.
Stage 3: The "Doctor's Rounds" Phase (Instruction Tuning)
- The Analogy: The intern is now shadowing senior doctors. They learn how to speak like a doctor, how to summarize a patient's history, and how to answer specific questions like, "Is this patient safe for surgery?"
- What happened: The model learned to follow complex instructions. It learned to write full medical reports, summarize findings, and answer tricky questions, mimicking the way human doctors think and communicate.
Stage 4: The "Coach's Whistle" Phase (Reinforcement Learning)
- The Analogy: The intern is now taking a final exam, but with a coach standing over them. Every time the intern points to the wrong spot or writes a wrong diagnosis, the coach gives a "ding" (a penalty). Every time they get it right, they get a "ding" (a reward).
- What happened: The model practiced thousands of times. If it drew a box around the wrong organ, the system said, "No, that's wrong." If it got the location right, it got a reward. This "coach" (Reinforcement Learning) fine-tuned the model to be incredibly accurate at spotting diseases and drawing boxes around them.

3. The Results: Why MedMO is a Game-Changer

The paper tested MedMO against other top medical AI models (like Fleming-VL and Lingshu) and even some closed-source giants (like Google's Gemini and OpenAI's GPT-4).

The "Eyes" Test: When asked to find bacteria in a microscope image or a fracture in an X-ray, MedMO was significantly better than everyone else. While other models were guessing or drawing boxes in the wrong place, MedMO was accurate.
The "Brain" Test: When asked to answer complex medical questions (like "What is the best treatment for this condition?"), MedMO scored higher than almost all open-source competitors.
The "Report" Test: When asked to write a medical report based on an image, MedMO wrote reports that sounded more like a real doctor and contained fewer errors.

4. The Big Picture

Think of MedMO as the first open-source "Master Diagnostician."

Before this, if you wanted a medical AI that could both see an image, point to the problem, and write a report, you had to use expensive, closed systems (like GPT-4) that you can't inspect or modify. MedMO is open-source, meaning any hospital, researcher, or developer can download it, study how it works, and use it to help patients.

In a nutshell:
MedMO is a medical AI that didn't just memorize textbooks; it went to medical school, learned to use a microscope, practiced drawing on X-rays, and was coached by a strict teacher until it could diagnose and point out problems better than almost any other open AI available today. It bridges the gap between "talking about medicine" and "actually understanding medical images."

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have achieved near-human performance in general domains, their application in medicine is hindered by three critical limitations:

Domain Gaps: General-purpose models often produce hallucinated or uncertain outputs when applied to medical tasks due to a lack of domain-specific contextualization and precise interpretation.
Distillation Reliance: Many existing medical MLLMs rely on distillation from proprietary models, which often lack accurate domain grounding and structured supervision, amplifying hallucination risks.
Narrow Modality Coverage: Current models typically focus on single modalities (e.g., only radiology or pathology) or individual tasks, failing to achieve unified, cross-modal generalization across the diverse imaging types found in real-world healthcare (e.g., MRI, CT, ultrasound, microscopy).
Weak Grounding: Existing models struggle with spatial reasoning, specifically the ability to localize abnormalities (bounding boxes) and link them to textual descriptions with high precision.

2. Methodology: MedMO

MedMO is an open-source, post-trained multimodal foundation model built upon the Qwen3-VL-8B architecture. It is designed to unify visual grounding, clinical reasoning, and language understanding. The model is trained via a four-stage progressive pipeline using a unified corpus of over 26 million samples from 45 diverse datasets.

Core Architecture

MedMO utilizes a Vision Encoder ( $E_v$ ), a Vision-Language Adapter ( $A$ ) featuring a DeepStack fusion mechanism to project multi-level ViT features into the language embedding space, and a Large Language Model Decoder ( $D$ ).

The Four-Stage Training Pipeline

Stage 1: General Medical SFT (Large-Scale Alignment)
- Data: 18.5M samples from the MedTrinity dataset (covering X-ray, CT, MRI, pathology, etc.).
- Objective: Establish foundational medical knowledge and global image-text alignment.
- Resolution: 768×768.
- Tasks: Captioning, VQA, and general multimodal tasks.
Stage 2: High-Resolution & Grounding SFT
- Data: 3M curated, expert-annotated samples including bounding-box annotations (Chest X-ray, Wrist X-ray, Cell Microscopy, CT).
- Objective: Enhance visual grounding and spatial awareness.
- Resolution: 1280×1280 (High-Resolution).
- Tasks: Object detection, referring expression comprehension, and spatial localization.
Stage 3: Instruction Tuning
- Data: 4.3M multimodal instruction-response pairs.
- Objective: Align model responses with human-style medical reasoning and instruction following.
- Tasks: Diagnostic Q&A, report summarization, and retrieval-based reasoning.
Stage 4: Reinforcement Learning (RL) with Verifiable Rewards
- Algorithm: GRPO (Group Relative Policy Optimization) adapted from DAPO.
- Reward Function: A composite reward system designed to verify factual accuracy and spatial precision:
  - Label Accuracy: Correctness of the diagnosis/text.
  - Bounding Box Reward: A novel GIoU-based reward using Hungarian matching to evaluate the overlap between predicted and ground-truth boxes. This directly optimizes spatial localization.
  - Tag Count & Soft Overlap Penalty: To prevent hallucination of non-existent objects.
- Objective: Refine instruction-following behavior and significantly boost spatial grounding accuracy.

3. Key Contributions

MedMO Model: A powerful, open-source post-trained VLM (available in 4B and 8B variants, with "Next" RL-tuned versions) specifically designed for comprehensive medical image understanding and grounding.
Unified Dataset Curation: Creation of a massive, harmonized corpus of 26M+ samples from 45 datasets, covering diverse modalities (Radiology, Pathology, Ophthalmology, Dermatology, Surgical Video) and biological systems.
Novel Grounding Benchmark: Construction of a dedicated Cell Dataset from open-source microscopy images (varying sizes, shapes, densities) to rigorously evaluate detection capabilities.
Verifiable RL Framework: Introduction of a bounding-box GIoU reward signal within the RL stage, providing a verifiable, spatially grounded signal that significantly improves localization performance over standard generative rewards.
Comprehensive Benchmarking: Extensive evaluation across VQA, Text QA, Report Generation, and Grounding tasks, establishing a new open benchmark for medical MLLMs.

4. Experimental Results

MedMO variants (specifically MedMO-8B-Next) achieve State-of-the-Art (SOTA) performance across multiple benchmarks, outperforming strong open-source competitors like Fleming-VL-8B and Lingshu-7B, as well as approaching proprietary models.

Visual Question Answering (VQA):
- MedMO-8B-Next achieves an average of 72.7% (vs. Fleming-VL-8B's 66.1%).
- Significant gains on MMMU-Med (+6.0%), PMC-VQA (+9.8%), and MedXpertQA (+21.3%).
Text-Based QA:
- Achieves 60.1% average (vs. 45.7% for Fleming-VL-8B).
- Leads on MedQA (83.8% vs. 66.1%) and MMLU-Med (80.2% vs. 71.8%).
Medical Report Generation:
- On MIMIC-CXR, MedMO-8B-Next achieves a CIDEr of 143.4 (vs. 132.5 for Fleming-VL-8B) and superior semantic fidelity (Semb: 51.5%).
- On Med-Trinity (multi-modal), it achieves a CIDEr of 272.1, more than 3x higher than the next best open-source model.
Grounding (Spatial Localization):
- Demonstrates a massive leap in localization capabilities.
- Bacteria Microscopy: Achieves 56.1 IoU (vs. 9.1 for Fleming-VL-8B), a gain of +47.0 IoU.
- DeepLesion: Achieves 40.5 IoU (vs. 0.00 for several baselines).
- MedSG Benchmarks: Consistently outperforms specialist benchmarks and general baselines in multi-view and object tracking tasks.

5. Significance

Open-Source Leadership: MedMO establishes the best fully open-source medical multimodal foundation model to date, providing a transparent and reproducible alternative to proprietary systems.
Robust Spatial Reasoning: By integrating a bounding-box reward into RL, MedMO solves a critical weakness in current MLLMs: the inability to precisely locate and describe medical abnormalities. This is crucial for clinical trust and diagnostic support.
Cross-Modality Generalization: The model successfully unifies understanding across radiology, pathology, and microscopy, moving beyond siloed medical AI solutions.
Scalable Roadmap: The paper provides a scalable training recipe (SFT + High-Res + Instruction + Verifiable RL) that future research can adopt to build reliable medical AI systems.

In conclusion, MedMO represents a significant step forward in medical AI by combining large-scale data curation with a specialized reinforcement learning strategy that prioritizes both factual correctness and precise spatial grounding.