Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Space

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have achieved significant success in medical image interpretation (e.g., classification, segmentation, and report generation), they lack robust visual-spatial intelligence in 3D medical imaging.

• The Gap: Current MLLMs struggle with quantitative spatial reasoning tasks essential for clinical practice, such as estimating organ volumes, calculating inter-structure distances, and understanding 3D topological relationships.
• The Bottleneck: Progress is hindered by a systemic lack of high-quality training and evaluation datasets. Existing datasets typically provide classification labels or narrative reports but lack structured 3D spatial annotations (e.g., precise Euclidean distances, volumetric magnitudes, and relative orientations) required for quantitative reasoning.
• The Consequence: Without grounded spatial data, models often rely on textual priors or "hallucinate" answers rather than performing genuine spatial analysis, posing safety risks for surgical planning and AI-assisted diagnostics.

2. Methodology: The SpatialMed Pipeline

To address the data scarcity, the authors propose an agentic pipeline to autonomously synthesize a high-quality benchmark dataset called SpatialMed.

A. Data Source

The pipeline curates data from multiple large-scale medical imaging datasets (TotalSegmentator, AMOS, Medical Segmentation Decathlon, KiTS, BraTS), resulting in 2,375 3D CT scans with corresponding segmentation masks covering 117 anatomical structures and 8 tumor types.

B. Agentic Synthesis Pipeline

The dataset generation involves three core stages:

1. Spatial Computational Tools: The system extracts precise quantitative metadata from CT scans using three specialized tools:
   • Volume Calculator: Computes volume by counting foreground voxels and converting using voxel spacing (mm³ to cm³).
   • 3D Bounding Box Extractor: Derives axis-aligned bounding boxes to define spatial extents.
   • Distance Calculator: Computes Euclidean distances between the centroids of bounding boxes for different structures.
2. Question Generation (RAG): A multi-agent system uses Retrieval-Augmented Generation (RAG) with PubMed as a knowledge source. Agents combine the extracted spatial metadata with medical knowledge to generate clinically meaningful Question-Answer (QA) pairs.
   • Task Types: Multiple-Choice Questions (MCA) covering direction, distance, extent, volume, and comparison; and direct Volume Estimation tasks.
3. Multi-Stage Validation:
   • Automated Filtering: A clinical validation agent and three medical specialist LLMs (InternLM 2, Qwen-3, Llama-3) filter out "trivial" questions (answerable via general knowledge without seeing the image) and check for medical plausibility.
   • Expert-in-the-Loop: Three board-certified radiologists manually review the remaining candidates. They score samples (0–2) based on Clinical Logicality, Quantitative Consistency, and Exhaustive Mutuality. Only samples with an average score > 1.0 are retained.

C. The Benchmark: SpatialMed

The final dataset comprises 9,782 QA pairs derived from 2,375 CT scans. It is designed to evaluate grounded, quantitative 3D spatial reasoning, distinguishing between answer correctness and reasoning faithfulness.

3. Key Contributions

1. Agentic Data Pipeline: A novel framework that combines computational geometry tools, multi-agent collaboration, and expert radiologist validation to automatically generate structured 3D spatial QA data.
2. SpatialMed Benchmark: The first comprehensive benchmark specifically for evaluating 3D spatial intelligence in medical MLLMs, covering diverse organs and tumor types with rigorous quantitative ground truth.
3. Comprehensive Evaluation: A systematic evaluation of 14 state-of-the-art MLLMs (including 2D models with/without medical pretraining and native 3D models), revealing critical performance gaps in spatial reasoning.

4. Experimental Results & Analysis

The authors evaluated 14 models (e.g., Qwen3-VL, InternVL3, MedFlamingo, M3D) on SpatialMed using Accuracy for MCA and Mean Relative Accuracy (MRA) for volume estimation.

• Overall Performance: Even top-performing models achieve only modest accuracy, often hovering near the random baseline (25%) for complex tasks.
• Key Bottlenecks:
  • Distance Reasoning: This is the most difficult task; even medically pre-trained models struggle to compare geometric relations accurately.
  • Volume Estimation: Models frequently fail to produce stable numerical outputs, often returning NaN or poorly calibrated values.
  • Scaling Issues: Increasing model size does not guarantee better spatial reasoning (e.g., Qwen3-VL 4B outperformed the 8B variant on volume tasks), suggesting sub-skill interference or lack of specific spatial training.
• Failure Analysis:
  • Error Types: The primary failure modes are Numeric Reasoning Errors (incorrect calculations) and Relational Reasoning Errors (misjudging positions/directions).
  • Hallucination: A significant portion of model outputs involves "hallucinated reasoning," where the chain-of-thought is incorrect even if the final answer happens to be right (lucky guesses), or both are wrong.
  • Organ/Tumor Variability: Performance is highly unstable across different anatomical structures; models that perform well on one organ often fail on another, indicating a lack of robust generalization.
• Medical Pretraining: Surprisingly, medical pretraining did not uniformly improve spatial reasoning. Models tuned for recognition and report generation often underperform on quantitative spatial tasks compared to general-domain models, suggesting that current medical tuning prioritizes semantic understanding over geometric operators.

5. Significance and Future Directions

• Clinical Impact: The study highlights that current MLLMs are not yet reliable for quantitative clinical tasks (e.g., surgical planning, tumor staging) where precise spatial measurements are critical.
• Paradigm Shift: The paper argues for moving beyond diagnostic classification toward spatially aware AI that can perform grounded, quantitative reasoning.
• Future Directions: The authors propose several avenues for improvement:
  • Developing spatial Chain-of-Thought reasoning.
  • Implementing self-supervised spatial objectives.
  • Enforcing structured numeric output constraints to improve calibration.
  • Adopting faithfulness-aware training to reduce hallucinations in spatial reasoning.

In summary, SpatialMed exposes a fundamental gap in the spatial reasoning capabilities of current medical AI, providing a necessary benchmark and dataset to drive the development of safer, more reliable, and truly spatially intelligent medical agents.