💬 NLP

UniReason-Med: A Shared Grounded Reasoning Interface for 2D-to-3D Transfer in Medical VQA

The paper introduces UniReason-Med, a unified framework that leverages a shared grounded reasoning interface and a large-scale 2D-3D instruction-tuning dataset (UniMed-CoT) to transfer reasoning capabilities from abundant 2D medical images to improve 3D medical visual question answering.

Original authors: Mengzhuo Chen, Yan Shu, Chi Liu, Hongming Piao, Xidong Wang, Derek Li, Bryan Dai

Published 2026-06-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mengzhuo Chen, Yan Shu, Chi Liu, Hongming Piao, Xidong Wang, Derek Li, Bryan Dai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to be a doctor. The robot needs to look at medical pictures (like X-rays or CT scans) and explain what it sees, pointing out exactly where problems are located.

The paper introduces a new system called UniReason-Med. Think of this system as a "universal translator" for medical reasoning that works for both flat, 2D pictures and thick, 3D volumes.

Here is the breakdown of how it works, using simple analogies:

1. The Problem: Two Different Languages

Doctors look at two very different types of images:

2D Images: Like a flat photograph of a chest X-ray.
3D Volumes: Like a loaf of bread (a CT scan). To look at it, you have to slice it into many thin slices and look at them one by one.

Previously, AI models were like students who only studied one subject. If you taught them on flat photos, they got confused by 3D bread-loaf scans. If you taught them only on 3D scans, they weren't as good at flat photos. Also, most AI just "guessed" the answer without showing its work or pointing to the specific spot on the image.

2. The Solution: A Shared "Pointing" System

The authors created a new way for the AI to "think" and "point" at the same time. They call this Grounded Chain-of-Thought (GCoT).

The Analogy: Imagine a teacher asking a student, "Where is the broken bone?"
- Old AI: Just says, "It's in the arm." (No proof).
- UniReason-Med: Says, "I see a break here [points to a box on the image], and because of that, I conclude it's a fracture."
The Magic: The system uses the same language (syntax) to point at a spot on a flat photo and a spot inside a 3D loaf of bread. It draws a box on the photo or a 3D cube around the slice. This allows the AI to use the "pointing" skills it learned from millions of 2D photos to help it understand 3D scans.

3. The Training: A Massive "Homework" Set

To teach this system, the researchers built a giant dataset called UniMed-CoT.

The Size: It contains 220,000 examples.
The Mix: 170,000 are flat 2D images, and 50,000 are 3D scans.
The Content: Each example isn't just a question and answer. It's a full "story" where the AI explains its reasoning step-by-step, drawing boxes around the evidence as it goes.
How they made it: They used an automated pipeline (like a smart robot assistant) to generate these stories and then had humans check a sample to make sure the "pointing" was accurate.

4. The Two-Step Learning Process

The AI learns in two stages, like a student taking a test and then getting extra credit:

Stage 1 (Supervised Fine-Tuning): The AI is shown the 220,000 examples and told, "Here is how you should think and point." It learns to mix text reasoning with visual pointing.
Stage 2 (Reinforcement Learning): This is the clever part. Usually, to teach a robot to point correctly, you need to give it a score based on how perfectly its drawn box overlaps with the real object (like a video game score).
- The Paper's Claim: The researchers did not give the AI these "overlap scores." Instead, they only rewarded the AI if the final answer was correct.
- The Result: Surprisingly, by just rewarding the correct final answer, the AI automatically got better at pointing accurately. It figured out that to get the right answer, it had to look at the right spot.

5. What They Found (The Results)

2D helps 3D: When they trained the AI on both 2D and 3D data together, the AI got much better at understanding 3D scans than if they had only trained it on 3D data. The "pointing" skills from the flat photos transferred over to the 3D bread-loaf scans.
Better than before: The system outperformed other medical AI models on standard tests for both 2D and 3D images.
No "Cheat Sheet" needed: The AI learned to point accurately without being explicitly graded on its drawing skills, just on its final diagnosis.

Summary

UniReason-Med is a single AI brain that can look at a flat X-ray or a 3D CT scan and explain its reasoning by "drawing" boxes around what it sees. By teaching it to do this for both types of images at the same time, the 2D knowledge helps the 3D understanding. Most importantly, it learned to point accurately just by trying to get the right answer, without needing a teacher to grade its drawings.

Note: The authors state this is for research and benchmarking purposes only, not for making real-life medical decisions yet.

Technical Summary: UniReason-Med

Problem Statement

Medical Multimodal Large Language Models (MLLMs) face a critical challenge in bridging the gap between 2D planar imaging (e.g., X-rays) and 3D volumetric data (e.g., CT, MRI). While physicians routinely synthesize evidence across these fundamentally different spatial representations, existing medical MLLMs typically operate in silos: early models focus on 2D instruction following, while newer 3D-capable models often lack explicit grounding mechanisms. Furthermore, current reasoning-oriented models rely primarily on textual Chain-of-Thought (CoT) without interleaving localized visual evidence, limiting interpretability and accuracy. There is a lack of a shared interface that enables cross-dimensional grounded transfer, specifically investigating whether abundant 2D grounded reasoning supervision can improve 3D medical Visual Question Answering (VQA) when both modalities share a unified reasoning protocol.

Methodology

The authors propose UniReason-Med, a single-checkpoint framework designed to process either a 2D image or a slice-serialized 3D volume using a shared Grounded Chain-of-Thought (GCoT) interface.

1. Unified Interface and GCoT

The core innovation is a shared language-side interface where reasoning trajectories interleave textual tokens with visual evidence extracted from specific regions.

Input Handling: 2D images are processed directly. 3D volumes are serialized into ordered sequences of 32 slices (following the M3D protocol) before visual encoding.
Grounding Syntax: The model generates coordinates to localize task-relevant regions. For 2D, it outputs bounding boxes $(x_1, y_1, x_2, y_2)$ . For 3D, it outputs cuboids $(x_1, y_1, z_1, x_2, y_2, z_2)$ , where the $z$ -axis corresponds to the slice index.
Visual Token Injection: Upon generating coordinates, the model crops the corresponding region (2D rectangle or 3D slice range) and injects the resulting visual tokens into the reasoning chain. This allows the model to reason over "what it sees" rather than relying solely on text.
Architecture: The framework utilizes a frozen Qwen2.5-VL vision tower and projector, with the contribution lying in the $z$ -aware grounded reasoning interface over slice-serialized inputs, not a native 3D volumetric encoder.

2. UniMed-CoT Dataset

To train this interface, the authors constructed UniMed-CoT, a 220K instruction-tuning dataset containing:

170K 2D samples derived from SAMed2D-v1.
50K 3D samples derived from M3D training cases.
Annotation Pipeline: Grounding coordinates are extracted directly from existing segmentation masks. GPT-4o is used to generate interleaved reasoning traces within <thought>...</thought> blocks, injecting special tokens for bounding boxes/cuboids at the first mention of a structure.
Quality Control: The dataset undergoes automated filtering (removing malformed coordinates or shallow reasoning) and manual auditing, achieving a 92% pass rate for correct grounding and coherent reasoning.

3. Two-Stage Training Paradigm

Stage 1: Supervised Fine-Tuning (SFT): The model is trained on UniMed-CoT using standard cross-entropy loss over discrete tokens (reasoning text, coordinates, and final answers). Continuous visual embeddings are injected but excluded from token-level supervision. This stage establishes the interleaved reasoning format.
Stage 2: Reinforcement Learning (RL): The model is further optimized using Group Relative Policy Optimization (GRPO). Crucially, the reward function is outcome-level (based on answer correctness and format adherence) and explicitly excludes IoU or Dice-based localization rewards. This tests whether optimizing for the final answer implicitly improves spatial grounding.

Key Contributions

Cross-Dimensional Grounded Transfer: The paper formulates and validates the hypothesis that abundant 2D grounded reasoning supervision can enhance 3D medical VQA when aligned through a shared language-side interface.
UniReason-Med Framework: A single-checkpoint model that unifies 2D images and slice-serialized 3D volumes via shared box syntax, region-token injection, and a grounded reasoning policy.
UniMed-CoT Dataset: A large-scale (220K) dataset with interleaved grounded CoT annotations for both 2D and 3D modalities, constructed via an automated pipeline with rigorous quality filtering.
Empirical Validation of RL without Localization Rewards: Demonstrating that outcome-level RL improves grounding performance without requiring explicit IoU/Dice rewards during the RL phase.

Experimental Results

The authors evaluate the model on OmniMedVQA (2D) and M3D-VQA (3D).

Cross-Dimensional Transfer (RQ1): Joint 2D+3D training significantly outperforms 3D-only training. On the 3D M3D-VQA benchmark, the mean accuracy increased from 61.2 (3D-only) to 70.2 (2D+3D) under matched training steps, confirming that 2D supervision benefits volumetric reasoning.
Component Ablation (RQ2):
- GCoT Interface: Moving from Text-CoT to Box-only GCoT, and finally to Full GCoT (with visual tokens), consistently improved performance in both 2D (62.0 $\to$ 65.3) and 3D (67.7 $\to$ 70.2).
- Training Stages: Combining SFT and RL yielded the best results (71.1 in 2D, 83.8 in 3D), surpassing SFT-only or RL-only configurations.
Grounding Improvement without Localization Rewards (RQ3): Despite the absence of IoU/Dice rewards during RL, the grounding quality improved significantly.
- 2D (Kvasir-SEG): Dice score improved from 0.54 (SFT) to 0.65 (SFT+RL).
- 3D (AMOS22/MSD): Dice scores improved from ~0.37 to ~0.43–0.45.
- Correlation: A strong positive correlation ( $r=0.832$ ) was observed between grounding accuracy and answer correctness, suggesting that optimizing for correct answers implicitly refines localization.

Significance and Claims

The paper claims that a shared grounded reasoning interface can successfully transfer reasoning structures from abundant 2D images to slice-serialized volumetric medical understanding. The significance lies in demonstrating that:

Joint Supervision is Superior: Training on a mixture of 2D and 3D data with a unified grounding syntax yields better 3D reasoning than 3D-only training.
Implicit Grounding is Possible: Outcome-level reinforcement learning can enhance spatial grounding and localization accuracy without the computational cost or data requirements of explicit localization rewards (IoU/Dice) during the RL phase.
Interpretability: The framework provides explicit visual evidence (via bounding boxes/cuboids and cropped tokens) alongside reasoning, addressing the "black box" nature of many medical MLLMs.

The authors maintain a modest stance, noting that the work is intended for research and benchmark analysis, not autonomous clinical decision-making, and that future work should explore denser volumetric encoders and prospective clinical validation.