💻 computer science

Can Experts Adapt Without Training? On Test-Time Modality Generalization in MVLMs

The paper proposes MoBE, a fully optimization-free framework that enhances the test-time modality generalization of medical vision-language models by combining entropy-guided dynamic expert routing with online Bayesian adaptation, thereby achieving superior accuracy across diverse medical benchmarks without requiring gradient updates.

Original authors: Raza Imam, Darakshan Rashid, Yutong Xie, Dwarikanath Mahapatra, Brejesh Lall, Mohammad Yaqub

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Raza Imam, Darakshan Rashid, Yutong Xie, Dwarikanath Mahapatra, Brejesh Lall, Mohammad Yaqub

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a spaceship navigating through a galaxy of medical images. Your ship is powered by a super-smart AI navigator, a "Medical Vision-Language Model" (MVLM). This navigator has studied millions of pictures of lungs, hearts, and bones, learning to recognize diseases with incredible speed. It's so good that it can often guess what's wrong just by looking at a new picture, even if it hasn't seen that exact type of scan before. This is called "zero-shot generalization," and it's the holy grail of medical AI.

However, space is full of surprises. Sometimes, the scanner changes, the lighting shifts, or a new type of imaging machine appears that the navigator never trained on. When this happens, the AI's confidence can crumble. It's like a chef who is a master at cooking Italian food but suddenly gets handed a menu of Thai dishes; they might guess, but they'll likely mess up. Scientists call this a "distribution shift." The big question is: Can we teach this AI to adapt instantly, while it's working, without sending it back to school for a long, expensive training course? This paper explores a clever way to let the AI "think on its feet" using a team of specialists who can swap roles and learn from each other in real-time, all without needing a single drop of new data or a computer restart.

The Problem: The "One-Size-Fits-All" Trap

In the world of medical AI, researchers have tried to solve the "new scanner" problem by building models with a "Mixture of Experts" (MoE). Imagine a hospital with a team of doctors, where each doctor is a super-specialist in one specific type of scan, like X-rays or MRIs. When a patient arrives, the system has to decide which doctor to listen to.

The paper points out a tricky dilemma here. If the system tries to listen to everyone at once (averaging all the doctors' opinions), the specific, sharp knowledge of the right specialist gets diluted by the noise of the others. But if the system blindly picks just one doctor who seems most confident, it might pick the wrong one if the patient's scan looks slightly different than expected. It's a classic catch-22: listening to everyone makes you average, but listening to one person makes you risky.

The Solution: MoBE (Mixture of Bayesian Experts)

The authors propose a new framework called MoBE. Think of MoBE not as a rigid robot, but as a dynamic, self-correcting team of detectives. Instead of retraining the whole team (which takes forever and requires massive data), MoBE lets the team adapt on the fly during the investigation. It does this in two playful but powerful steps:

1. The "Entropy" Detective (Dynamic Routing)
First, the system needs to figure out which specialists are actually paying attention. It uses a concept called "entropy," which is a fancy word for "confusion." If a specialist is very confused about an image, their entropy is high. If they are confident, their entropy is low.
MoBE acts like a smart manager who doesn't just pick the top 3 experts. Instead, it looks at the team and says, "Whoever is almost as confident as the most confident person gets to speak." It filters out the confused experts and keeps a small, reliable group. This is called dynamic-k routing. It's like a sports coach who doesn't just pick the star player, but picks the whole squad that is currently "in the zone," ensuring the team is diverse but focused.

2. The "Bayesian" Memory (Expert Adaptation)
Once the right experts are chosen, they need to tweak their thinking for the specific patient in front of them. Usually, AI models are frozen; they can't change their minds once trained. MoBE breaks this rule without breaking the model.
Each expert carries a tiny, invisible "memory bank" (a Bayesian posterior). As they look at new images, they update this memory.

The Prototype Update: If an expert sees a clear image of a "broken bone," they update their mental picture of what a "broken bone" looks like to match this new, slightly different image.
The Prior Update: They also update their "gut feeling" about how common certain diseases are in this new environment.
Crucially, they only update their memory if they are very confident. It's like a detective who only writes down a new clue if they are 99% sure it's real, preventing them from getting confused by fake leads.

The Results: Smarter Without the Homework

The authors tested this "training-free" approach on a massive collection of medical datasets, including X-rays, MRIs, CT scans, and even microscopic tissue samples. They compared MoBE against other top-tier methods that usually require heavy computer power to retrain the model on the fly.

The results were impressive. Without changing a single weight in the main brain of the AI (no "backpropagation" or gradient updates), MoBE managed to boost accuracy significantly:

On standard medical datasets, it improved average accuracy by +4.72% over the best existing methods.
On datasets with completely unseen types of scans, it jumped by +7.17%.
On a chaotic mix of different medical images, it gained +4.3%.

For example, on a dataset called BreastMNIST, MoBE achieved a stunning 73.08% accuracy, far beating the previous best of 52.56%. Even on tricky datasets where the AI usually struggles, like BloodMNIST, it showed it could handle the chaos better than its competitors.

The Catch and the Future

There is a small price to pay for this magic. Because MoBE has to run multiple "experts" (doctors) in parallel to decide who is confident, it takes a bit more time to process each image. The paper notes that while it is slower than a single frozen model, it is much faster than methods that try to retrain the model while working. It's the difference between asking one person to think really hard versus asking a whole team to think quickly and vote.

The authors conclude that this "route-and-adapt" strategy proves you don't need to retrain an AI to make it robust against new medical scanners. By letting experts dynamically select themselves and update their own confidence levels, we can build medical AI that is not just smart, but also adaptable and ready for the real world's surprises. While the method isn't perfect (it still struggles a bit when the new scans are too different from anything the experts have ever seen), it offers a promising, lightweight path forward for making medical AI safer and more reliable in the clinic.

Technical Summary: MoBE – Test-Time Modality Generalization in Medical Vision-Language Models

1. Problem Statement

Medical Vision-Language Models (MVLMs) demonstrate strong zero-shot generalization capabilities but suffer from reliability collapse when deployed on unseen modalities, scanners, or acquisition protocols. While Mixture-of-Experts (MoE) architectures offer a mechanism to leverage modality-specific knowledge, their application at test time faces a specialization-generalization dilemma:

Blind Aggregation: Uniformly averaging all experts dilutes modality-specific knowledge.
Single-Expert Reliance: Selecting a single "most confident" expert risks catastrophic mismatch if that expert's confidence is misplaced under distribution shift.

Existing Test-Time Adaptation (TTA) methods often rely on gradient-based optimization (e.g., MoME, TNT) or single-model memory caches (e.g., TDA), which either require computationally expensive backpropagation or fail to address the specific challenge of selecting among multiple modality-specialized experts without source data. The core challenge is how to dynamically route and adapt experts at inference time without any parametric updates or access to training data.

2. Methodology: MoBE (Mixture of Bayesian Experts)

The authors propose MoBE, a fully optimization-free framework that integrates dynamic expert routing with per-expert Bayesian adaptation. The system operates on a frozen MVLM backbone (e.g., BiomedCLIP) augmented with parallel, modality-specialized MLP experts.

A. Dynamic-k Entropy-Guided Expert Routing

To resolve the specialization-generalization dilemma, MoBE employs a Dynamic-k routing strategy based on predictive uncertainty rather than fixed thresholds or single-expert selection.

Entropy Calculation: For each test sample, predictive entropy $H(p_e)$ is computed across all $E$ experts. Lower entropy indicates higher modality alignment.
Relative Thresholding: Instead of selecting a fixed top- $k$ , the method sorts experts by ascending entropy ( $H_1 \le H_2 \le \dots \le H_E$ ) and selects a dynamic set $E_k$ where the entropy gap relative to the best expert is within a modality-agnostic threshold $\tau$ :
$E_k = \{e_j \mid H_j - H_1 < \tau\}$
This ensures the selection includes experts that are nearly as confident as the best one while excluding clearly mismatched ones.
Inverse-Entropy Weighting: Selected experts are weighted using inverse entropy ( $w_e \propto 1/\exp(H(p_e))$ ) rather than softmax, which better captures relative confidence differences.

B. Expert Bayesian Adaptation (EBA)

Once experts are routed, MoBE adapts their predictions online without updating backbone or expert weights. Each selected expert maintains two persistent memory variables updated via exponential moving averages (EMA) across the test stream:

Class Prototypes ( $\mu_{e,c}$ ): Moving representatives of class features, initialized from pretrained text embeddings.
Class Priors ( $\Pi_e$ ): A distribution capturing the expert's historical tendency to predict specific classes.

Confidence-Gated Updates:

Prototype Update: Triggered when the posterior confidence exceeds a threshold $\theta_1$ . The prototype for the predicted class is updated by averaging it with the current input feature.
Prior Update: Triggered at a higher confidence threshold $\theta_2$ . The class prior is updated using the current posterior probability.
Final Prediction: The system blends the entropy-routed logits with the Bayesian-adapted logits ( $s_{final} = (1-\lambda)s + \lambda s_{EBA}$ ), allowing the model to align with the target distribution while preserving pretrained semantic structure.

3. Key Contributions

Identification of the Dilemma: The paper reveals the "specialization-generalization dilemma" that hinders test-time modality adaptation in Medical VLMs, where static fusion or single-expert reliance fails under distribution shifts.
MoBE Framework: Introduction of a fully optimization-free inference framework that unites dynamic expert routing with per-expert Bayesian adaptation, enabling online confidence updates and prototype alignment without backpropagation.
Empirical Validation: Demonstration of consistent state-of-the-art gains across standard (MedMNIST, Med-VTAB) and heterogeneous medical benchmarks, proving that training-free expert specialization enables robust modality generalization.

4. Experimental Results

The authors evaluated MoBE against state-of-the-art TTA methods (TENT, EATA, TPT, TDA, MoME) and prompt-learning baselines (CoOp, CoCoOp) across 11 datasets spanning 9 imaging modalities (X-ray, CT, MRI, Ultrasound, Pathology, etc.).

Seen Modalities: On the combined MedMNIST+Med-VTAB benchmark, MoBE achieved an average accuracy of 43.41%, outperforming the baseline BiomedCLIP (33.00%) and the optimization-based MoME (38.69%). Notable gains were observed on BreastMNIST (73.08%).
Unseen Modalities: In scenarios where test modalities were unseen during expert pre-training, MoBE significantly improved average accuracy to 38.16%, compared to 20.49% for BiomedCLIP and 30.99% for MoME.
Heterogeneous Benchmarks: On a diverse suite of 11 datasets covering 10 organs and 9 modalities, MoBE achieved the highest average accuracy (46.49%), surpassing TDA (42.19%).
Efficiency: Unlike optimization-based methods (e.g., MoME) that require backpropagation, MoBE is inference-only. While it incurs a higher computational cost than a single frozen model due to multiple forward passes, it avoids the heavy overhead of gradient-based adaptation (e.g., MoME takes ~13 minutes vs. MoBE's ~11 minutes for the same dataset, but MoBE achieves higher accuracy).

5. Significance and Claims

The paper claims that MoBE effectively addresses the reliability gap in clinical deployment by enabling MVLMs to adapt to distribution shifts without requiring retraining or source data access. By framing the problem through the lens of expert routing and Bayesian adaptation, the authors demonstrate that:

Training-free adaptation is viable: Robust generalization can be achieved without gradient updates, making the approach suitable for environments where backpropagation is infeasible.
Dynamic routing is superior to static selection: The entropy-guided dynamic-k approach outperforms fixed top-k selection and uniform fusion, effectively balancing the trade-off between specialization and generalization.
Online statistics matter: Maintaining and updating class prototypes and priors allows experts to adapt to the evolving test distribution, correcting for centroid misalignment and prior shifts.

The authors acknowledge that MoBE incurs higher inference latency than a single frozen model due to the need for multiple expert forward passes. They suggest future work should focus on reducing this overhead through techniques like early-exit routing, pruning, or distillation, and improving robustness when the available expert bank does not fully cover the test distribution.