Technical Summary: CT-SpatialVQA Benchmark for 3D Medical VLMs

Problem Statement

Recent advances in 3D medical Vision-Language Models (VLMs) have demonstrated proficiency in generating fluent descriptions and answering semantic or measurement-based questions regarding volumetric medical images. However, a critical uncertainty remains: whether these models possess genuine semantic-spatial reasoning capabilities grounded in 3D anatomy or if they merely rely on learned language priors and statistical correlations found in radiology reports.

Current benchmarks and training paradigms often fail to systematically evaluate spatial grounding—the ability to infer precise anatomical localization, laterality, structural comparisons, and 3D inter-structure relationships across volumetric data. Existing datasets (e.g., DeepTumorVQA) focus heavily on descriptive queries, while standard 3D VLMs, even those with 3D backbones, lack explicit mechanisms to enforce cross-slice consistency or geometric inductive biases. Consequently, models may exhibit linguistic fluency while failing at clinically essential spatial judgments, such as determining if a lesion is anterior or posterior to a vessel, or distinguishing left from right laterality.

Methodology

1. The CT-SpatialVQA Benchmark

To address the evaluation gap, the authors introduce CT-SpatialVQA, a benchmark specifically designed to test semantic-spatial reasoning in 3D CT data.

• Data Source: The benchmark is constructed from the CT-RATE dataset, comprising 1,601 unique radiology reports paired with 3D CT volumes.
• QA Generation Pipeline: A robust, multi-stage pipeline was employed to generate 9,077 clinically grounded question-answer (QA) pairs:
  1. Generation: An LLM (GPT-4o) was prompted to extract spatial observations from radiology reports and generate QA pairs. The prompts strictly limited the scope to six spatial dimensions: laterality (left/right), longitudinal position (superior/inferior), anterior-posterior relations, medial-lateral orientation, adjacency/containment, and spatial extent. Diagnostic or severity-based content was explicitly excluded.
  2. Validation: An independent Validator LLM (Gemini 2.5 Flash) reviewed candidate pairs against the source report to verify that answers were strictly derived from the text, free of hallucinations, and focused on spatial attributes.
  3. Human Audit: A random subset (12%) of validated pairs underwent human review by trained reviewers. This audit confirmed a 95% consensus agreement rate, validating the reliability of the automated pipeline.
• Evaluation Protocol: Models are tested in a zero-shot setting. They receive a 3D CT volume and a spatial question but no radiology report. This ensures the model must reason directly from the volumetric visual data rather than relying on textual cues.

2. Evaluation Metrics

The study evaluates eight state-of-the-art 3D medical VLMs (including CT-Chat, MERLIN, Med3DVLM, M3D, RadFM, VILA-M3, MedGemma-1.5, and MedEvalKit) using a dual-track assessment:

• LLM-as-Jury: Three independent LLMs (GPT-4o, Gemini 2.5 Flash, Qwen3) act as judges to determine binary correctness, with an aggregated "Jury" decision serving as the primary accuracy metric.
• Traditional Metrics: Semantic alignment is measured via SBERT cosine similarity, while lexical overlap is assessed using BLEU, ROUGE-L, and METEOR.

Key Results

The evaluation reveals a severe degradation in the performance of current 3D medical VLMs on semantic-spatial reasoning tasks:

• Low Accuracy: All evaluated models scored below 50% accuracy. The average accuracy across models was 34.08%, with the best-performing model (CT-Chat) achieving only 43.69%. Several models performed at or below random chance levels.
• Semantic Misalignment: The average SBERT cosine similarity was 0.51, and lexical metrics (BLEU: 4.81, ROUGE-L: 0.22) were modest, indicating that generated answers frequently diverged from ground truth both semantically and lexically.
• Spatial Dimension Variance: Performance varied across spatial categories. Models showed sharper declines in vertically and depth-oriented reasoning (e.g., superior/inferior, anterior/posterior) compared to central-peripheral localization. This suggests a specific failure in maintaining consistent volumetric representations across depth.
• Failure Modes: The results indicate that models often rely on prior knowledge or language correlations rather than performing genuine volumetric visual grounding.

Significance and Claims

The paper claims that CT-SpatialVQA establishes spatial grounding as a measurable and benchmarkable property, rather than an assumed byproduct of scaling multimodal models. The significance of this work lies in:

1. Exposing a Critical Gap: It demonstrates that linguistic fluency in medical VLMs does not equate to true spatial understanding, raising concerns about their reliability for high-stakes clinical decision support where geometric precision is essential.
2. Diagnostic Framework: The benchmark provides a diagnostic tool to identify specific failure modes (e.g., inability to handle cross-slice consistency) in current training paradigms.
3. Future Direction: The findings suggest that future 3D medical VLMs must move beyond surface-level image-text correlations. The authors argue for architectures that explicitly encode volumetric structure and spatial consistency to achieve trustworthy clinical utility.

The authors conclude that without addressing these structural constraints in training objectives, 3D medical VLMs will remain insufficient for tasks requiring fine-grained, spatially grounded reasoning.