💻 computer science

Adapting Foundation Vision-Language Models to Medical Diagnosis via Query-Driven Expert Bridging

MedBridge is a lightweight, model-agnostic framework that adapts foundation vision-language models for medical diagnosis by employing query-driven expert bridging to simultaneously address domain shifts, resolution mismatches, and multi-label reasoning, achieving significant performance improvements across diverse benchmarks.

Original authors: Yitong Li, Morteza Ghahremani, Christian Wachinger

Published 2026-05-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yitong Li, Morteza Ghahremani, Christian Wachinger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, world-class art critic who has spent their entire life studying famous paintings in museums. They can instantly recognize a Van Gogh or a Picasso just by looking at a brushstroke. This critic is like a Foundation Vision-Language Model (VLM)—a powerful AI trained on billions of everyday images and text.

Now, imagine you ask this art critic to diagnose a patient's chest X-ray. They might try to apply their knowledge of paintings to the X-ray, but they fail miserably. Why? Because X-rays are high-resolution, black-and-white medical images full of tiny, specific details (like a tiny nodule) that look nothing like colorful paintings. If you shrink the X-ray down to fit the critic's standard viewing size (like squinting at a tiny photo), those tiny details vanish, and the diagnosis is lost.

The paper introduces MedBridge, a clever "translator" and "manager" that helps this art critic become a skilled medical diagnostician without forcing them to go back to school for years.

Here is how MedBridge works, using simple analogies:

1. The Problem: The "Blurry Photo" and the "Wrong Library"

The Resolution Issue: Standard AI models are used to looking at small, 224x224 pixel images. Medical X-rays are huge (2048x2048). If you shrink an X-ray to fit the model, it's like taking a high-definition photo of a fingerprint and shrinking it until it's a blur. The AI misses the tiny clues needed for a diagnosis.
The Domain Gap: The AI knows about cats, cars, and landscapes (natural images). It doesn't know about pneumonia or broken bones (medical images). It's like asking a chef who only cooks Italian food to suddenly cook a complex Japanese banquet.
The Multi-Label Challenge: A single X-ray often has multiple problems at once (e.g., a patient might have both an enlarged heart and fluid in the lungs). Most AI models are trained to pick just one answer, but a doctor needs to spot all of them.

2. The Solution: MedBridge

MedBridge is a lightweight "adapter" that sits between the AI and the X-ray. It doesn't retrain the whole AI (which is expensive and slow); instead, it adds a few smart tools to help the AI see and think differently.

A. The "Magnifying Glass" (Multi-View Sampling)

Instead of showing the AI the whole shrunk-down X-ray, MedBridge acts like a magnifying glass. It takes the huge X-ray and cuts it into several smaller, high-resolution "patches" (like looking at different corners of a map).

How it helps: This ensures the AI doesn't miss tiny, subtle clues (like a small tumor) that would disappear if the whole image were shrunk down. It looks at the "forest" (the whole image) and the "trees" (the detailed patches) simultaneously.

B. The "Smart Sticky Notes" (Learnable Query Tokens)

The paper introduces "learnable queries." Imagine the AI is a library with millions of books (its training data). MedBridge adds a set of smart sticky notes to the books.

How it works: These notes don't rewrite the books (the AI's original knowledge stays frozen and safe). Instead, the notes ask the AI specific questions like, "Does this look like a lung infection?" or "Is there fluid here?"
The Magic: These notes are "learnable," meaning they get smarter as the AI practices on medical data. They act as a bridge, translating the AI's general knowledge into medical language without destroying what the AI already knows.

C. The "Team Captain" (Mixture of Experts)

MedBridge doesn't rely on just one AI model. It acts like a team captain who can call on different specialists.

How it works: It can connect to different types of AI experts (some good at general images, some good at medical reports). The "Smart Sticky Notes" (the queries) act as a signal to the captain: "Hey, for this specific X-ray, we need the advice of Expert A and Expert B."
The Result: The AI combines the best insights from different models to make a final decision, even if those models were trained on totally different things.

3. The Results: Why It Matters

The authors tested MedBridge on five different medical datasets (thousands of chest X-rays).

Better Accuracy: It improved diagnosis accuracy by 6% to 15% compared to other methods. That's a huge jump in the medical world.
Fast and Cheap: Unlike other methods that require massive computers and weeks of training, MedBridge can adapt an AI in a few hours on a single computer.
Flexible: It works with many different types of AI models (like CLIP, LLaVA, or Qwen), proving it's a universal tool, not a one-trick pony.

Summary

Think of MedBridge as a universal translator and a magnifying glass for AI. It takes a general AI that knows about the world, gives it a magnifying glass to see tiny medical details, and hands it a set of smart notes that teach it how to ask the right medical questions. It lets the AI become a doctor's assistant quickly, accurately, and without needing to be rebuilt from scratch.

Technical Summary: MedBridge – Adapting Foundation Vision-Language Models to Medical Diagnosis

1. Problem Statement

While foundation Vision-Language Models (VLMs) have achieved significant success in natural image classification, their direct application to medical imaging faces three critical, mutually reinforcing constraints that existing adaptation methods fail to address jointly:

Resolution Mismatch and Information Loss: Medical images (e.g., chest X-rays) are typically high-resolution (~2048 × 2048) and contain sparse, spatially localized pathological evidence. Standard VLMs require aggressive downsampling (e.g., to 224 × 224), causing subtle but critical diagnostic features (such as small nodules) to vanish.
Severe Domain Shift: There is a substantial discrepancy between the visual statistics and label semantics of natural images and medical images. Directly perturbing pretrained weights via full fine-tuning risks destroying the generalizable source knowledge, while standard adaptation strategies often fail to bridge this gap without catastrophic forgetting or overfitting, especially in data-scarce clinical settings.
Multi-Label and Heterogeneous Nature: Medical diagnosis is inherently multi-label, where a single image may contain multiple co-existing diseases indicated by distinct, non-overlapping visual cues (e.g., global structural changes for cardiomegaly vs. fine local edge patterns for pneumothorax). A single foundation model often struggles to capture this diverse diagnostic evidence effectively.

Current approaches either require costly, data-intensive pre-training of medical-specific foundation models or rely on adaptation methods that do not simultaneously handle resolution preservation, domain alignment, and multi-label reasoning.

2. Methodology: MedBridge

The authors propose MedBridge, a lightweight, model-agnostic adaptation framework that transforms frozen foundation VLMs into multi-view query encoders augmented with a query-guided Mixture-of-Experts (MoE) routing mechanism. The framework operates through two primary modules:

A. Multi-view Query Encoder (MQEncoder)

Instead of processing a single downsampled image, the MQEncoder employs a multi-view focal sampling strategy:

Input Augmentation: From a high-resolution input image, the system extracts $N-1$ local regions (via random selection or sliding window) and stacks them with the resized global image to create an $N$ -view input. This preserves fine-grained pathological cues and reduces sensitivity to precise image alignment.
Non-Destructive Alignment: The backbone VLM encoder remains frozen. Learnable query tokens are injected directly into intermediate layers (starting from layer $n_1 + 1$ ) of the frozen encoder.
Interaction: These queries interact with the frozen tokens via Multi-Head Self-Attention (MHSA) and Feed-Forward Networks (FFN). The queries serve a dual role:
1. Adapting the frozen backbone to medical semantics by bridging the domain gap.
2. Generating routing representations for the MoE module.
Fusion: After processing, tokens are aggregated across views. A final prediction token is formed by fusing the frozen tokens ( $\bar{m}$ ) and the learnable query tokens ( $\bar{q}$ ) via a weighted sum: $p_k = \bar{m}_k + \alpha \cdot \bar{q}_k$ , where $\alpha$ is a hyperparameter controlling the contribution of the learned adaptation.

B. Query-Driven Mixture of Experts (MoE)

Unlike traditional MoE mechanisms that route based on input tokens, MedBridge's MoE is driven by the learnable query tokens:

Routing Mechanism: A gating network takes the concatenated learnable queries from $K$ heterogeneous expert VLMs as input. It computes a weight vector $g(Q_G)$ via a softmax layer, dynamically weighting the contribution of each expert.
Heterogeneous Integration: This allows the integration of diverse foundation models (e.g., CLIP, LLaVA, Qwen-VL, MedGemma) that do not share a common representation space. The queries act as a unified interface to aggregate complementary diagnostic cues from different experts.
Final Prediction: The final vision embedding is a weighted sum of the prediction tokens from all experts.

C. Training Objective

The system uses a multi-label binary cross-entropy (BCE) loss. The vision embedding (from the MoE) is compared against text embeddings of disease labels (encoded as "a radiology image with [CLASS]") using cosine similarity. The model is optimized to push similarities for present diseases toward 1 and absent ones toward 0.

3. Key Contributions

Query-Driven Adaptation Interface: The introduction of MQEncoders that inject learnable queries into frozen backbones, enabling non-destructive domain alignment while preserving fine-grained pathology through multi-view sampling.
Flexible Query-Conditioned MoE Routing: A novel routing mechanism where learnable queries serve as both adaptation variables and routing signals, enabling the dynamic integration of heterogeneous VLMs without requiring a shared representation space.
Robust Empirical Gains: Demonstration of superior performance across five medical datasets and three adaptation tasks (cross-domain, in-domain, and zero-shot), showing significant improvements over state-of-the-art methods with minimal parameter overhead (approx. 4% increase).

4. Experimental Results

The authors evaluated MedBridge on five chest radiograph benchmarks (MIMIC-CXR, CheXpert Plus, NIH ChestX-ray14, RSNA Pneumonia, COVIDx CXR-4) and extended the evaluation to retinal fundus imaging (ODIR-5K).

Cross-Domain Adaptation: When adapting general-purpose VLMs (e.g., CLIP) to medical data, MedBridge achieved a 6–15% AUC improvement over state-of-the-art methods (e.g., CLIP-LoRA, MMRL). For instance, on CheXpert Plus, MedBridge (w/ MoE) reached 83.55% AUC, outperforming the next best method (MMRL at 76.46%) by over 7%.
In-Domain Adaptation: For models pre-trained on medical data (e.g., CheXzero), MedBridge showed strong performance on both seen and unseen diseases. Notably, for the unseen disease (COVID-19), it achieved 78.1% AUC, outperforming the second-best method by 5%.
Zero-Shot Cross-Dataset: MedBridge demonstrated strong generalization, achieving 79.94% AUC on CheXpert Plus after being adapted on MIMIC-CXR, surpassing the next best method by nearly 3.5%.
Model Agnosticism: The framework successfully adapted diverse VLMs (CLIP, LLaVA, Qwen3-VL, MedGemma), with performance scaling as the size and specialization of the backbone increased.
Efficiency: MedBridge completes adaptation in a few hours on a single GPU, contrasting with the days required for full medical pre-training. It also demonstrated lower memory usage and training time compared to LoRA-based alternatives when used as a drop-in replacement.
Generalization: The method maintained robust performance on retinal fundus imaging (ODIR-5K) without modality-specific tuning, achieving 72.58% AUC.

5. Significance and Claims

The paper claims that MedBridge represents a new direction in domain-gap mitigation for medical imaging. Its significance lies in:

Solving the Triad of Constraints: It uniquely addresses resolution mismatch, domain shift, and multi-label heterogeneity simultaneously, which previous methods have not done jointly.
Non-Destructive Adaptation: By keeping foundation models frozen and using a compact set of learnable queries, it preserves the generalizable knowledge of large-scale pre-training while adapting to specific clinical needs.
Practical Utility: It offers a computationally efficient, model-agnostic solution that can rapidly repurpose arbitrary foundation models for multi-label disease diagnosis, making it highly suitable for clinical environments with limited data and computational resources.
Synergistic Design: The authors emphasize that the performance gains stem from the synergistic interaction of multi-view sampling, query injection, and heterogeneous expert routing, rather than any single component alone.

The authors conclude that MedBridge effectively bridges the gap between general-purpose foundation models and the rigorous demands of medical diagnosis, providing a flexible and powerful tool for clinical applications.