💻 computer science

M-IDoL: Information Decomposition for Modality-Specific and Diverse Representation Learning in Medical Foundation Model

The paper proposes M-IDoL, a self-supervised medical foundation model that employs information decomposition via Mixture-of-Experts to maximize inter-modality entropy and minimize intra-modality uncertainty, thereby achieving superior modality-specific and diverse representations that outperform existing models across 21 downstream clinical tasks.

Original authors: Yihang Liu, Ying Wen, Jiaxiong Yang, Longzhen Yang, Lianghua He, Heng Tao Shen

Published 2026-04-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yihang Liu, Ying Wen, Jiaxiong Yang, Longzhen Yang, Lianghua He, Heng Tao Shen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a single, super-smart student to become a doctor. This student needs to learn how to read five different types of medical books:

X-rays (looking at bones and lungs).
Eye scans (looking at the retina).
Skin photos (looking for moles).
Microscope slides (looking at cells).
Eye tomography (looking at eye layers).

The Problem: The "Confused Student"

In the past, researchers tried to teach this student by throwing all five types of books at them at once. They said, "Just learn the general idea of 'medicine' from all these pictures."

The problem? The student got confused.

When looking at a skin mole, the student started thinking about lung bones.
When looking at an eye scan, they started mixing it up with skin textures.

Because the student tried to force all these different pictures into one single "mental filing cabinet," the details got blurry. They learned the common stuff (like "this is a picture") but forgot the special stuff (like "this specific texture means skin cancer"). This is what the paper calls "Information Ambiguity." The student became a generalist who wasn't very good at being a specialist.

The Solution: M-IDoL (The "Smart Librarian" System)

The authors of this paper created a new teaching method called M-IDoL. Instead of forcing the student to mix everything together, they gave them a Smart Librarian (called a "Mixture-of-Experts" or MoE).

Here is how M-IDoL works using a simple analogy:

1. The Sorting Hat (Inter-Modality Specificity)

Imagine the Smart Librarian has five different colored desks:

🟥 Red Desk for X-rays.
🔵 Blue Desk for Eye scans.
🟢 Green Desk for Skin photos.
🟡 Yellow Desk for Microscope slides.
🟣 Purple Desk for Eye tomography.

When a picture comes in, the Librarian immediately asks: "What kind of picture is this?" and sends it to the correct colored desk.

The Goal: Make sure the X-ray student never sits at the Skin desk, and the Skin student never sits at the X-ray desk.
The Result: Each student becomes a true expert in their own field because they aren't distracted by the other types of books. This is called Maximizing Inter-Modality Entropy (making the groups very distinct).

2. The Deep Dive (Intra-Modality Diversity)

Once the picture is at the correct desk (say, the Skin desk), the student doesn't just look at it once. They look at it from every angle, zoomed in, zoomed out, and flipped.

They ask: "Is this mole dangerous? Is it just a shadow? Is it a scar?"
They learn the tiny, subtle details that make one skin condition different from another.
The Goal: Make sure the student learns everything about skin, so they can tell the difference between a harmless freckle and a dangerous melanoma.
The Result: The student becomes incredibly detailed and precise. This is called Minimizing Intra-Modality Uncertainty (removing the confusion within the specific topic).

Why is this a big deal?

Previous models were like a student trying to read a physics textbook and a poetry book at the same time, hoping to understand "words." They ended up understanding neither well.

M-IDoL is like having a team of specialists where:

The Librarian ensures the right expert gets the right book (so X-rays don't get mixed with Skin).
The Expert studies their specific book so deeply they can spot the tiniest details (so they can find a tiny tumor).

The Results

The researchers tested this new system on 1.15 million medical images (a huge library!). They then asked the student to solve 21 different medical puzzles (like diagnosing glaucoma or finding pneumonia).

The Outcome:

The M-IDoL student beat 20 other top medical AI models.
It was better at spotting diseases in X-rays, eyes, skin, and cells than models that were trained only on those specific types of images.
It proved that you can have one AI that is a master of everything, as long as you teach it to keep its "specialist" skills separate and sharp.

In a Nutshell

M-IDoL is a new way to train medical AI. Instead of mashing all medical images into one big, blurry soup, it uses a "sorting hat" to keep different types of images separate, allowing the AI to become a super-specialist in each field while still being part of one powerful team. This leads to more accurate diagnoses and better patient care.

1. Problem Statement

Medical Foundation Models (MFMs) aim to learn universal representations from multimodal medical images (e.g., X-ray, Fundus, OCT, Pathology) to generalize across diverse clinical tasks. However, existing unified MFMs suffer from information ambiguity.

The Core Issue: Current models typically use a unified Contrastive Learning (CL) objective that maximizes Mutual Information (MI) between augmented views of images from different modalities within a single embedding space.
The Consequence: This approach forces the model to learn "modality-shared redundancy" (common features across all modalities) rather than modality-specific features. It blurs the boundaries between different imaging types (e.g., confusing the texture of a skin lesion with a lung nodule) and fails to capture fine-grained semantic details unique to each modality.
The Goal: The paper seeks to resolve this ambiguity by learning representations that are both modality-specific (distinct clusters for different imaging types) and intra-modality diverse (fine-grained discrimination within a single imaging type).

2. Methodology: M-IDoL

The authors propose M-IDoL, a self-supervised foundation model that introduces Information Decomposition to explicitly separate modality-specific information from shared redundancy.

A. Theoretical Framework: Information Decomposition

The standard CL objective maximizes $I(X; Y)$ , where $X$ and $Y$ are augmented views. In a multimodal setting, this inadvertently maximizes $I(X; Y; Z)$ (trivariate MI), where $Z$ represents representations from other modalities. This term represents the unwanted shared redundancy.

M-IDoL reformulates the objective to maximize intra-modality information while minimizing the interference from other modalities:
$\max I(X; Y) - I(X; Y; Z)$

Using information theory, the authors decompose this into two entropy-based objectives:

Maximize Inter-modality Entropy ( $H(X|Z)$ ):
- Goal: Ensure that the representation $X$ of a specific modality contains information that cannot be predicted by representations $Z$ from other modalities.
- Effect: This forces the model to disperse different modalities into distinct subspaces, enhancing modality specificity.
Minimize Intra-modality Uncertainty ( $H(X|Y, Z)$ ):
- Goal: Reduce the predictive uncertainty of $X$ given its augmented view $Y$ , conditioned on $Z$ .
- Effect: This encourages the model to focus on reducing noise caused by data augmentation (e.g., cropping, color jitter) rather than semantic shifts between modalities, thereby enhancing intra-modality diversity and fine-grained discrimination.

B. Architectural Implementation: MoE Projector

To implement these objectives without prior knowledge of modality labels (since training data is shuffled), M-IDoL employs a Mixture-of-Experts (MoE) Projector after the visual encoder.

Structure: The projector consists of a Router and $N$ Experts (where $N$ corresponds to the number of modalities, e.g., 5).
Routing Mechanism: The router assigns each image to a specific expert based on learned probabilities.
Loss Functions:
1. Routing-Consistency Loss ( $L_{route}$ ):
  - Enforces that augmented views of the same image are routed to the same expert.
  - Uses the Sinkhorn-Knopp algorithm to ensure a balanced assignment of images across experts, preventing all images from collapsing into a single expert.
  - Purpose: Pushes different modalities into distinct MoE subspaces, maximizing $H(X|Z)$ .
2. Intra-modality Contrastive Loss ( $L_{cst}$ ):
  - Applied within each expert's subspace. It pulls together representations of augmented views of the same image routed to that expert.
  - Purpose: Minimizes $H(X|Y, Z)$ by learning fine-grained features specific to that modality's subspace.

The total loss is: $L_{M-IDoL} = L_{route} + L_{cst}$ .

3. Key Contributions

Information Decomposition Strategy: The paper introduces a novel theoretical framework to decompose multimodal mutual information, explicitly removing modality-shared redundancy to mitigate information ambiguity in unified pre-training.
MoE-based Architecture: A self-supervised MoE projector is designed to adaptively learn latent modality subspaces without requiring explicit modality labels during training. It simultaneously optimizes for inter-modality separation and intra-modality diversity.
Comprehensive Evaluation: The model is pre-trained on 1.15 million unlabeled medical images across five major modalities (Retinology, Radiology, Ophthalmology, Histopathology, Dermatology) and evaluated on 21 downstream clinical tasks.

4. Experimental Results

Performance: M-IDoL consistently outperforms 20 existing foundation models (including SOTA unified models like UniMed, LVM-Med, CoSMIC, and modality-specific models like RETFound and Virchow) across 21 downstream datasets.
- It achieves significant improvements (p < 0.05) in tasks such as diabetic retinopathy grading, glaucoma detection, and skin lesion classification.
- It outperforms the second-best unified model by over 6% in several tasks.
Ablation Studies:
- Removing the MoE projector or the routing loss ( $L_{route}$ ) leads to performance drops, confirming the necessity of separating modalities.
- Removing the contrastive loss ( $L_{cst}$ ) reduces fine-grained discrimination within modalities.
Visualization (t-SNE):
- Unified Models: Show blurred clusters where different modalities overlap significantly (information ambiguity).
- M-IDoL: Displays sharp separation between different modalities (inter-modality specificity) while maintaining tight, well-structured clusters within each modality (intra-modality diversity).

5. Significance

Paradigm Shift: M-IDoL challenges the prevailing "one-size-fits-all" unified pre-training approach. It demonstrates that treating medical imaging modalities as a single homogeneous distribution degrades performance.
Efficiency: Unlike previous methods that use continual learning (multi-stage training) to achieve modality specificity, M-IDoL achieves this in a single-stage unified pre-training process, making it computationally more efficient.
Clinical Impact: By preserving fine-grained semantic details specific to each imaging type (e.g., subtle pigment irregularities in dermatology vs. faint opacities in radiology), M-IDoL provides a more robust foundation for diverse clinical diagnostic tasks, potentially improving the reliability of AI-assisted diagnosis across the medical spectrum.