🤖 AI

Adapting 2D Multi-Modal Large Language Model for 3D CT Image Analysis

This paper proposes a method to adapt 2D Multi-Modal Large Language Models for 3D CT image analysis by reusing pre-trained parameters and introducing a Text-Guided Hierarchical MoE framework with a two-stage training strategy to effectively handle 3D medical tasks like report generation and visual question answering.

Original authors: Yang Yu, Dunyuan Xu, Yaoqian Li, Xiaomeng Li, Jinpeng Li, Pheng-Ann Heng

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yang Yu, Dunyuan Xu, Yaoqian Li, Xiaomeng Li, Jinpeng Li, Pheng-Ann Heng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant, world-class art critic (the AI) how to understand 3D medical scans like CTs and MRIs.

Here is the problem: This critic has spent their entire life studying 2D paintings and photographs. They are amazing at describing a flat picture of a cat. But now, you hand them a giant, 3D block of cheese (a 3D medical scan) and ask them to describe the holes inside it or answer specific questions about a tiny crumb. They get confused because they've never seen a 3D object before, and they don't have enough practice books (data) to learn from scratch.

This paper proposes a clever solution to fix this. Here is how it works, broken down into simple steps:

1. The "2D to 3D" Adaptation (Teaching the Critic to See Depth)

Instead of trying to build a new critic from scratch (which is hard because there aren't enough 3D medical books), the authors take their existing, super-smart 2D critic and give them a special pair of glasses.

The Old Way: Previous researchers tried to build a new "3D-only" brain. But because there are very few 3D medical images in the world compared to the billions of 2D photos on the internet, this new brain is weak and hasn't learned much.
The New Way: They take the strong 2D brain (which already knows how to see textures and shapes) and modify its "lenses." They teach it to look at the 3D scan not as one giant block, but as a stack of 2D slices (like looking through a loaf of bread). They tweak the brain's internal layers so it can understand the depth of the loaf, not just the surface of a single slice.
The Result: The AI gets the "street smarts" of the 2D world but learns how to navigate the "3D world" without forgetting what it already knew.

2. The "Text-Guided Hierarchical MoE" (The Smart Switchboard)

Now that the AI can see the 3D scan, there is a second problem. Doctors use these scans for two very different jobs:

Writing a Report (MRG): "Describe everything you see in this scan." (Needs a broad, global view).
Answering a Question (MVQA): "Is there a tumor in the left kidney?" (Needs a laser-focused view on just one spot).

Previous AI models used the same "brain mode" for both. It's like using a wide-angle camera lens to take a selfie; you get the whole room, but the face is blurry.

The authors built a Smart Switchboard called TGH-MoE:

The Text Clue: The AI looks at the doctor's question (the text prompt). If the text says "Write a report," the switchboard knows to use the "Broad View" experts. If the text says "Where is the tumor?", it switches to the "Laser Focus" experts.
The Hierarchy: It's like a two-level management team.
- Level 1 (Task Manager): Decides which team to use (Report Team vs. Question Team).
- Level 2 (Specialists): Inside that team, there are many tiny specialists who each look at a small part of the image to build the final answer.
The Analogy: Imagine a restaurant. If you order a "Full Tasting Menu" (Report), the chef uses all their skills to prepare a full meal. If you order "Just the Soup" (Question), the chef focuses entirely on making the perfect soup. The AI does the same thing automatically based on your order.

3. The Two-Stage Training (The "Generalist then Specialist" Strategy)

You can't just flip the switch and expect the AI to be perfect immediately. If you try to teach it to be both a generalist and a specialist at the same time, it gets confused and learns poorly.

So, the authors use a Two-Stage Training Camp:

Stage 1 (The Generalist Phase): They turn off the "Task Switchboard." They force the AI to look at both reports and questions using the same "brain mode." This teaches the AI the shared basics of medical images (what a kidney looks like, what a tumor looks like) without getting distracted by the specific task.
Stage 2 (The Specialist Phase): Now that the AI knows the basics, they turn the "Task Switchboard" back on. They tell the AI, "Okay, now that you know what a kidney is, learn how to describe the whole kidney for reports, and how to spot a tumor for questions."
The Result: The AI learns the foundation first, then specializes, leading to much better performance.

Why This Matters

Better Diagnosis: The AI can now write more accurate medical reports and answer specific questions better than any previous 3D AI.
Efficiency: It doesn't need to be trained from scratch on scarce medical data; it reuses the massive knowledge of 2D internet images.
Versatility: It handles the "big picture" and the "tiny details" equally well, just like a human radiologist does.

In short: The authors took a smart 2D AI, taught it to see in 3D, gave it a smart switch to change its focus depending on the task, and trained it in two steps to become the ultimate medical assistant.

1. Problem Statement

The paper addresses the challenges in applying Multimodal Large Language Models (MLLMs) to 3D medical image analysis, specifically for Medical Report Generation (MRG) and Medical Visual Question Answering (MVQA). Two primary limitations exist in current state-of-the-art (SOTA) 3D medical MLLMs:

Insufficient Perceptual Capacity: Existing 3D MLLMs typically pre-train 3D Vision Transformers (ViTs) from scratch using limited 3D medical image-text datasets. Due to the scarcity of 3D medical data (orders of magnitude smaller than natural image datasets), these encoders lack the robust feature extraction capabilities found in models pre-trained on massive 2D natural image-text corpora.
Lack of Task-Specific Adaptability: Current 3D MLLMs extract the same image features for all tasks. However, MRG requires perceiving global, holistic image features to generate descriptive reports, while MVQA often requires focusing on specific, localized regions relevant to a question. Treating these tasks identically leads to suboptimal performance.

2. Methodology

The authors propose a novel framework that adapts a well-trained 2D MLLM to handle 3D volumetric inputs while introducing a mechanism to distinguish between different tasks. The methodology consists of three core components:

A. Adapting 2D ViT for 3D Volumetric Inputs

Instead of training a 3D ViT from scratch, the authors modify a pre-trained 2D ViT (from a 2D MLLM) to process 3D medical volumes ( $D \times H \times W$ ) while reusing all pre-trained parameters.

Patch Embedding: The 3D volume is treated as a sequence of 2D slices. The input is reshaped to fit the 2D patch embedding layer (e.g., treating depth $D$ as a sequence of 3-slice groups).
Positional Embedding: Since 2D positional embeddings cannot distinguish depth ( $D$ ), Rotary Position Embeddings (RoPE) are added to the self-attention queries and keys to encode 3D spatial positions.
Attention Mechanism: Lower layers (specialized in local 2D textures) remain as 2D attention blocks. Higher layers are modified into 3D attention blocks by reshaping inputs to capture volumetric context.
Patch Merging: Standard 2D patch merging is used for $H$ and $W$ , followed by average pooling along the $D$ dimension to reduce token count.
Fine-tuning: Unlike methods that freeze the vision encoder, this approach updates the encoder's parameters during fine-tuning to adapt to 3D medical data.

B. Text-Guided Hierarchical Mixture of Experts (TGH-MoE)

To address the need for task-specific features, the authors integrate a TGH-MoE framework into the vision encoder's Feed-Forward Networks (FFNs).

Task-Level MoE: A router uses the text prompt (which indicates the task type, e.g., "generate a report" vs. "answer a question") to select between specific experts. A hard router is used to binarize weights, assigning the entire image to either an MRG expert or an MVQA expert, preventing optimization interference.
Token-Level MoE: Within each task-specific expert, a token-level MoE routes individual image tokens to a subset of sub-experts. This creates a sparse, powerful network capable of capturing fine-grained details for specific tokens.
Hierarchy: This two-level structure allows the model to share low-level features (via the initial routing) while diverging into specialized high-level representations based on the task.

C. Two-Stage Training Strategy

Directly optimizing TGH-MoE leads to poor performance because the model fails to learn shared representations before specializing. The authors propose a two-stage strategy:

Stage 1 (Task-Shared Learning): The task-level MoE is disabled. Only the Token-level MoE is active, forcing both MRG and MVQA tasks to share the same image features. The entire network (Vision Encoder, Connector, LLM) is fine-tuned using auto-regressive loss.
Stage 2 (Task-Specific Learning): The Token-level MoE trained in Stage 1 is replicated to initialize the experts for the Task-Level MoE. The LLM is frozen, and only the Vision Encoder and Connector are fine-tuned. The router is trained to distinguish tasks, allowing the model to learn task-discriminative features while retaining the shared foundation.

3. Key Contributions

2D-to-3D Adaptation: A strategy to adapt powerful, pre-trained 2D MLLMs for 3D medical volumetric analysis, inheriting strong perceptual capacity from massive natural image pre-training without requiring new learnable parameters for the base architecture.
TGH-MoE Framework: A novel Text-Guided Hierarchical MoE that dynamically extracts task-specific image features (MRG vs. MVQA) based on text prompts, addressing the variability in feature requirements across tasks.
Two-Stage Training Strategy: A training protocol that effectively balances the learning of task-shared and task-specific features, overcoming the optimization interference common in multi-task MoE training.
State-of-the-Art Performance: The method achieves superior results on both MRG and MVQA tasks compared to existing 3D medical MLLMs (e.g., M3D, Med3DVLM).

4. Experimental Results

The method was evaluated on the M3D-Data dataset (containing ~106k caption samples and ~438k VQA samples).

Medical Visual Question Answering (MVQA):
- Closed-ended: Achieved 81.06% accuracy, outperforming the second-best method (M3D at 79.35%) by 1.71%.
- Open-ended: Surpassed M3D with improvements of 2.54% in BLEU-1 and 2.78% in ROUGE-1.
Medical Report Generation (MRG):
- Achieved the best scores in ROUGE-1 and METEOR, indicating higher textual quality and clinical completeness.
- Outperformed the 14B parameter model (RadFM) in 8 out of 9 metrics while using only 4B parameters.
Ablation Studies:
- Confirmed that the 2D-to-3D adaptation is critical for utilizing 3D spatial information.
- Demonstrated that the Two-Stage Training is essential; skipping Stage 1 (learning shared features first) resulted in suboptimal MRG performance.
- Showed that the Hard Router in TGH-MoE is superior to soft routers, as it prevents task interference.

5. Significance

This paper bridges the gap between the abundance of 2D natural image data and the scarcity of 3D medical data. By successfully transferring the perceptual power of 2D MLLMs to 3D domains and introducing a mechanism to handle task diversity, the authors provide a more efficient and effective solution for clinical AI assistants. The approach demonstrates that specialized task handling (via MoE) and strategic training (two-stage) are just as critical as the architecture itself for medical multimodal tasks. The method sets a new benchmark for 3D medical image analysis, offering a path toward more accurate automated diagnosis and reporting systems.