Technical Summary: DeepTumorVQA

Problem Statement

Medical Vision-Language Models (VLMs) and AI agents have advanced from 2D image understanding to 3D volumetric reasoning and tool-augmented diagnostic workflows. However, existing evaluation benchmarks suffer from two critical limitations:

1. Fragmentation: Benchmarks cover disjoint parts of the diagnostic problem (e.g., 2D recognition vs. 3D volume) rather than a unified protocol.
2. Aggregate Scoring: Most benchmarks reduce performance to a single accuracy score, obscuring where and why models fail. A model might answer a high-level diagnostic question correctly via dataset bias or language priors without grounding its answer in low-level visual evidence (e.g., recognizing a lesion but failing to measure its volume).

This lack of granularity prevents researchers from distinguishing between failures in visual perception, quantitative measurement, tool orchestration, and clinical reasoning.

Methodology

The authors propose DeepTumorVQA, a hierarchical 3D CT benchmark designed to decompose tumor diagnosis into a multi-stage evidence chain.

1. Benchmark Design

• Scale: The benchmark comprises 476,000 questions across 42 clinical subtypes derived from 9,262 abdominal 3D CT volumes sourced from 17 public datasets (88 medical centers).
• Hierarchical Structure: Questions are organized into four increasing levels of complexity, mirroring the clinical evidence chain:
  1. Recognition (9 subtypes): Atomic detection of organs and lesions (e.g., "Is there a kidney cyst?").
  2. Measurement (5 subtypes): Quantitative extraction of properties like organ volume, Hounsfield Unit (HU) values, and lesion counts.
  3. Visual Reasoning (16 subtypes): Compositional tasks combining recognition and measurement with spatial comparisons (e.g., "Which kidney is larger?" or "Count lesions in segment 2").
  4. Medical Reasoning (12 subtypes): Integration of measured evidence with clinical knowledge and guidelines (e.g., "Is there fatty liver based on the liver-to-spleen HU ratio?").
• Compositional Generation: Questions are generated deterministically from structured metadata (segmentation masks, HU values) using logic programs. This ensures ground-truth answers are derived from precise measurements rather than manual annotation of the final answer, allowing for the creation of "tool traces."

2. Evaluation Protocols

DeepTumorVQA supports two primary evaluation modes:

• Direct Inference: Models answer questions directly from image inputs (2D slices, 3D volumes, or video) without external tools.
• Tool-Augmented Agents: Models operate in a ReAct-style loop, calling external tools to gather evidence before answering. The benchmark provides three controlled environments:
  • Oracle Mode: Tools return ground-truth values from expert segmentation masks (upper bound).
  • Predicted Mode: Tools return values from automatic segmentation (e.g., TotalSegmentator), simulating realistic deployment.
  • Vision Mode: Models access only image crops and knowledge lookups, with no quantitative measurement tools.

3. Experimental Setup

The authors evaluated over 30 model configurations, including:

• 3D Specialists: M3D-Phi3, M3D-Llama2, Merlin, RadFM (3D-finetuned and zero-shot).
• 2D VLMs: Qwen3-VL, Meissa, LLaVA-Med, InternVL3, HuatuoGPT (zero-shot and LoRA-finetuned).
• Agents: Various configurations of the above models equipped with diagnostic tools (segmentation, measurement, knowledge lookup).
• Human Baselines: Junior and senior radiologists evaluated on a stratified subset.

Key Results

1. The Measurement Bottleneck

The most significant finding is that reliable quantitative measurement is the primary bottleneck for medical VLMs.

• Zero-shot models perform reasonably on Recognition (50–58%) but collapse on Measurement (24–32%, near random chance).
• This measurement failure cascades into Visual and Medical Reasoning, as these tasks require comparing measured values against thresholds.
• Even 3D specialists trained on volumetric data struggle with Measurement without task-specific fine-tuning.

2. Tool Augmentation and Agent Performance

• Tools Unlock Reasoning: Providing oracle tools (ground-truth measurements) dramatically improves performance. For example, Medical Reasoning accuracy jumps from ~27% to ~51–54% when models have access to correct measurements.
• New Failure Modes: While tools solve the measurement bottleneck, they introduce tool-use failures. Zero-shot agents frequently enter "action loops" (redundant tool calls) or fail to invoke necessary knowledge tools (used in only ~1.3% of calls despite being required for 19% of questions).
• Trace Supervision: Supervising agents with ground-truth tool-use traces (Step-by-Step SFT) resolves these orchestration failures. It reduces action loops from ~18% to 0.2% and increases overall agent accuracy from 46% to 63.8%.

3. 2D vs. 3D and Fine-tuning

• Task Supervision Dominates: 2D models fine-tuned on the benchmark's evidence primitives (e.g., Meissa-FT) outperform the best 3D specialists (66.3% vs. 59.8%), despite using only 2D inputs. This suggests that aligning the model with the specific diagnostic task and answer space is more critical than the input modality (3D volume vs. 2D slices).
• Video Inputs: Video inputs provided marginal gains (+1–2%) for some models but degraded performance for others, likely due to attention dilution across frames without explicit 3D positional encoding.

4. Causal Analysis

• Necessity of Measurement: Removing measurement tools causes a 16.2% drop in overall accuracy and a 24.1% drop in Medical Reasoning.
• Sufficiency of Measurement: For capable backbones (e.g., Qwen3.5-9B), providing only pre-computed measurements (no image) yields 54.4% accuracy, surpassing the full oracle agent (48.5%). This indicates that for structured diagnostic tasks, quantitative evidence is a sufficient statistic for reasoning, provided the model has the arithmetic and clinical knowledge capabilities.
• Precision Sensitivity: Even 10% noise in measurement values causes a catastrophic drop in Measurement accuracy (from 65% to 40%), highlighting the non-negotiable need for precise quantitative tools.

Significance and Contributions

The paper claims three main contributions:

1. DeepTumorVQA Benchmark: A large-scale, hierarchical 3D CT benchmark with 476K questions that moves beyond aggregate scoring to provide stage-wise diagnostic evaluation. It allows researchers to pinpoint whether a model fails at recognition, measurement, or reasoning.
2. Unified Evaluation Protocol: A framework that evaluates both direct VLM inference and tool-augmented agents across 2D, 3D, and video modalities, including controlled ablations (Oracle, Predicted, Vision) to isolate specific capability failures.
3. Diagnostic Insights: The benchmark reveals that quantitative measurement is the causal bottleneck in 3D CT VQA. It demonstrates that while tools can mitigate this, the ability to orchestrate tools and reason over correct evidence remains a challenge. Furthermore, it shows that trace supervision is effective in teaching agents efficient workflows, shifting the failure mode from tool-use errors to pure reasoning errors.

The authors conclude that DeepTumorVQA provides a concrete roadmap for future research, shifting the focus from "ranking models" to "understanding failure modes" and "evidence-grounded evaluation." All data and code are released to facilitate reproducibility.