RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering

1. Problem Statement

Radiologic Visual Question Answering (VQA) is a critical task for evaluating the clinical reasoning capabilities of Vision-Language Models (VLMs). However, existing medical VQA datasets suffer from three major limitations:

• Scale and Modality Bias: Most datasets are small-scale or dominated by X-ray images and biomedical illustrations, lacking sufficient representation of Computed Tomography (CT) and Magnetic Resonance Imaging (MRI), which are central to modern radiology.
• Linguistic Shortcuts: Many benchmarks allow models to achieve high accuracy by exploiting textual priors or distributional biases (e.g., guessing "pneumonia" because it appears frequently in the text) without actually interpreting the visual content.
• Evaluation Rigor: Current metrics for free-text report generation often fail to capture clinical correctness, while existing VQA benchmarks lack the granularity to test fine-grained pathology identification.

There is a pressing need for a large-scale, expert-curated dataset covering CT and MRI that forces models to rely on visual grounding rather than text-based shortcuts.

2. Methodology

The authors introduce RadImageNet-VQA, a dataset constructed from the CT and MRI subsets of the RadImageNet dataset. The construction pipeline involves:

• Data Source: Utilization of expert-annotated CT and MRI images from RadImageNet, covering 8 anatomical regions and 97 pathology categories.
• Sample Generation:
  • Radiologic Captioning: Metadata (modality, anatomy, pathology) is converted into structured, clinically meaningful captions using diverse templates to support visual-text alignment.
  • VQA Generation: 7.5 million question-answer pairs are generated across three task families:
    1. Anatomy Recognition: Identifying the imaged body region.
    2. Abnormality Detection: Determining if an abnormality exists (Yes/No).
    3. Pathology Identification: Distinguishing specific diseases within an anatomical context (the most challenging task).
  • Question Formats: The dataset includes Open-ended, Closed-ended (Yes/No), and Multiple-Choice (MC) questions.
• Mitigation of Shortcuts:
  • Distractor Design: For MC questions, distractors are carefully selected to be clinically plausible within the same anatomical region, preventing models from solving questions based solely on anatomy.
  • "No Pathology" Option: Every pathology MC question includes an option for "no pathology seen" to prevent models from assuming pathology is always present.
  • Template Diversity: Multiple linguistic variations are used for the same semantic query to reduce reliance on specific phrasing.
• Benchmark Construction: A stratified test set of 1,000 images (9,000 QA pairs) was created, preserving the distribution of the training set while ensuring balanced representation across tasks and modalities.

3. Key Contributions

1. RadImageNet-VQA Dataset: The release of the largest radiologic VQA dataset to date, featuring 750,000 CT/MRI images paired with 7.5 million QA samples. It is the first to comprehensively cover CT and MRI with fine-grained pathology labels.
2. Rigorous Benchmark: A curated evaluation set designed to minimize linguistic shortcuts, validated through text-only ablation studies showing that models drop to near-random accuracy without image inputs.
3. Comprehensive Evaluation: Extensive zero-shot and fine-tuning experiments on state-of-the-art general-purpose (e.g., InternVL, Qwen2.5-VL, GPT-5) and medical-oriented models (e.g., MedGemma, Lingshu).
4. Empirical Insights:
   • Demonstration that while anatomy recognition is nearly solved, fine-grained pathology identification remains a major bottleneck for all current VLMs.
   • Evidence that medical pretraining of vision encoders (e.g., MedSigLIP) does not necessarily outperform general-purpose encoders when fine-tuned on this specific radiologic task.
   • Confirmation that fine-tuning on RadImageNet-VQA yields substantial performance gains across all model families.

4. Key Results

• Zero-Shot Performance:
  • Anatomy: High performance (e.g., InternVL3.5-8B achieves ~93% on MC anatomy questions).
  • Abnormality Detection: Mixed results; some general models (GPT-5, Gemini 2.5 Pro) performed poorly, potentially due to safety alignment suppressing abnormality detection.
  • Pathology Identification: The primary failure mode. Even the best models scored below 31% on open-ended pathology questions. Fine-grained distinctions (e.g., specific ligament injuries) were largely missed.
• Text-Only Analysis:
  • On existing datasets (VQA-RAD, SLAKE), models achieved 11–33% accuracy without images.
  • On RadImageNet-VQA, text-only accuracy collapsed to 2–10% (near random), confirming the dataset effectively eliminates linguistic shortcuts.
• Fine-Tuning:
  • Fine-tuning on the RadImageNet-VQA corpus resulted in average accuracy improvements of +19.5% to +22.5%.
  • Anatomy recognition reached near-ceiling performance (~99%).
  • Abnormality detection saw large gains (+16% to +40%).
  • Pathology identification improved but remained the limiting factor.
  • Surprising Finding: Models initialized with general-purpose vision encoders (SigLIP) performed equally well or better than those initialized with medically pretrained encoders (MedSigLIP), suggesting that the volume and quality of the fine-tuning data outweigh the benefits of specific medical pretraining for this task.

5. Significance

RadImageNet-VQA represents a significant step forward in medical AI evaluation and development:

• Benchmarking Standard: It provides a rigorous, large-scale benchmark that forces models to demonstrate true visual grounding rather than text-based guessing, addressing a critical flaw in previous datasets.
• Training Resource: The dataset serves as a scalable resource for instruction tuning, proving that large-scale CT/MRI supervision can significantly boost the clinical reasoning capabilities of general VLMs.
• Clinical Relevance: By focusing on CT and MRI and fine-grained pathology, it aligns more closely with real-world radiological workflows than X-ray-centric datasets.
• Future Direction: The results highlight that while current models are good at recognizing anatomy, the next frontier in medical VLMs is fine-grained visual reasoning for subtle and complex pathologies, requiring better visual grounding mechanisms rather than just larger language models or medical pretraining.

The dataset and benchmark are publicly available at https://huggingface.co/datasets/raidium/RadImageNet-VQA.