An Automated, Contamination-Controlled VQA Benchmark for Evaluating Vision-Language Models on 3D Oncology Imaging

Technical Summary: An Automated, Contamination-Controlled VQA Benchmark for Evaluating Vision-Language Models on 3D Oncology Imaging

Problem Statement
Vision-language models (VLMs) are increasingly applied to medical imaging, yet their reliability in clinical settings remains poorly characterized. Existing public benchmarks for medical visual question-answering (VQA) suffer from two critical flaws:

1. Data Contamination: Large-scale datasets often enter the pretraining corpora of VLMs, meaning high scores may reflect memorization of image-question pairs rather than genuine visual perception.
2. Text-Only Solvability: Many benchmark items can be answered correctly using only the question text and language priors, bypassing the need to analyze the image.

Furthermore, oncology imaging presents unique challenges: studies are volumetric (3D), requiring interpretation across multiple slices and sequences, whereas most current VLMs accept only 2D inputs. Existing benchmarks fail to distinguish between image-dependent performance and performance driven by dataset familiarity or question-text cues.

Methodology
The authors present an automated, agent-driven pipeline designed to generate contamination-controlled multiple-choice VQA benchmarks directly from paired private radiology reports and 3D oncology imaging.

• Data Source: The pipeline utilizes private, single-institution radiology reports and 3D imaging (CT and MRI) from four oncology cohorts: liver CT, liver MRI, lung CT, and brain MRI (2,509 cases total for the main benchmark). A fifth, fully private in-house brain MRI cohort was generated identically but reserved specifically for the public/private contamination ablation study (Table 3) and excluded from the main benchmark totals. Because these source reports are private and not part of public pretraining data, the generated questions are guaranteed to be free from instance-level contamination.
• Question Generation: The system produces two complementary question types:
  1. Schema-Driven (RADS-style): Questions are populated deterministically from clinician-reviewed schemas grounded in established reporting frameworks (LI-RADS, Lung-RADS, and a RANO-aligned brain schema). This ensures standardized, visual-answerable questions.
  2. Report-Derived: Questions are generated by a language model based on the radiologist's findings and impression, then filtered by a "report-grounded check" to ensure the answer is explicitly supported by the source text.
• Image Processing: For 2D-input models, 3D volumes are rendered as 12×12 composite montages of 144 axial slices (approx. 128×128 pixels per slice).
• Evaluation Protocol: Five contemporary VLMs were evaluated in a zero-shot setting: two frontier closed models (Claude Opus 4.6, GPT-5.2), one open-weights frontier model (Qwen3-VL-30B), and two medical specialists (MedGemma1.0-27B, MedGemma1.5-4B).
• Ablation Studies: To isolate visual reliance, models were re-evaluated with the image replaced by a blank input ("blind" condition). Additionally, a public brain cohort (PDGM) was compared against a matched private in-house brain cohort to test for image-distribution familiarity.
• Statistical Rigor: The study employs case-clustered bootstrapping for confidence intervals and multiplicity-controlled tests (Benjamini-Hochberg FDR correction and Two One-Sided Tests [TOST] for equivalence) to distinguish genuine effects from noise.

Key Results

• Variable Performance: No single model was reliable across all cohorts. Accuracies ranged from near-random baseline (0.28) to 0.81. Model rankings varied significantly by cohort and question type.
• Image Independence: For several high-performing configurations, removing the image had little effect on accuracy.
  • On Brain MRI (both public and private) and Lung CT, frontier models (e.g., Claude Opus 4.6) showed minimal accuracy drops when images were removed, suggesting their performance relies heavily on language priors or question-text signals rather than visual evidence.
  • Conversely, MedGemma1.0-27B showed significant drops on Liver CT and MRI when blinded, indicating stronger visual reliance in those specific contexts.
• Public vs. Private Brain Imaging: On RADS-style questions, frontier models scored 0.17–0.19 higher on public (PDGM) brain images compared to matched private images, suggesting potential familiarity with public datasets. However, on report-derived questions, performance was indistinguishable between public and private cohorts.
• Task Difficulty: Models struggled significantly with quantitative or graded judgments (e.g., lesion size, Lung-RADS category, mass-effect grading), where accuracies barely exceeded chance. In contrast, qualitative recognition tasks (e.g., presence/absence of findings) were performed with much higher accuracy.
• Statistical Findings: After FDR correction, 37 of 40 accuracy measurements exceeded chance levels. However, equivalence testing revealed that for many high-scoring models on specific tasks (e.g., Brain MRI report-derived), the difference between sighted and blind performance was statistically equivalent to zero within a ±0.05 margin.

Significance and Claims
The paper does not claim to provide a definitive model leaderboard but rather demonstrates that a renewable, privately generated benchmark can effectively separate image-dependent performance from question-text and dataset-familiarity effects.

• Clinical Implications: Current VLMs, including medical specialists, are not yet reliable readers of volumetric oncology studies when presented in reduced 2D formats. Headline accuracies may overstate visual competence, particularly for tasks solvable via language priors (e.g., lung CT, brain MRI).
• Task Limitations: The inability of models to handle spatial measurements (size, severity grading) compared to simple recognition suggests that current 2D-input systems are ill-suited for management-driving tasks that rely on precise quantification.
• Methodological Contribution: The authors release the pipeline as an open agent skill, enabling institutions to regenerate contamination-controlled benchmarks on their own private data. This shifts evaluation from static public artifacts to private, renewable in-house tests, addressing the critical issue of data leakage in medical AI evaluation.

The study concludes that while VLMs show promise, their visual reasoning capabilities in complex, volumetric oncology settings remain limited and highly dependent on the specific imaging modality and question type. The proposed benchmark offers a rigorous framework to identify these limitations and prevent the deployment of models that rely on memorization rather than perception.