⚡ electrical engineering

NeuroQA: A Large-Scale Image-Grounded Benchmark for 3D Brain MRI Understanding

NeuroQA is a large-scale, image-grounded benchmark comprising nearly 57,000 question-answer pairs derived from 3D brain MRI volumes across 12 datasets and five clinical domains, designed to rigorously evaluate 3D medical visual reasoning while eliminating text-only shortcuts through strict image-grounding protocols and expert verification.

Original authors: Mohammad H. Abbasi (Stanford University), Favour Nerrise (Stanford University), Shaurnav Ghosh (Stanford University), Ridvan Yesiloglu (Stanford University), Yuncong Mao (Stanford University), Bailey

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohammad H. Abbasi (Stanford University), Favour Nerrise (Stanford University), Shaurnav Ghosh (Stanford University), Ridvan Yesiloglu (Stanford University), Yuncong Mao (Stanford University), Bailey Trang (Stanford University), Mohammad Asadi (Stanford University), Merryn Daniel (Stanford University), Gustavo Chau Loo Kung (Stanford University), Ken Chang (Stanford University), Pavan Pinkesh Shah (Stanford University), Adam Turnbull (Stanford University), Kyan Younes (Stanford University), Seena Dehkharghani (Stanford University), Ehsan Adeli (Stanford University)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to read a brain scan. The robot is incredibly smart; it has read millions of medical textbooks and knows the names of every part of the brain. But there's a catch: when you show it a 3D brain scan and ask, "Is there a tumor here?", the robot might not actually be looking at the scan. Instead, it might just be guessing based on the words in your question or the name of the patient group it was trained on. It's like a student who memorized the answer key to a test without ever studying the material.

This paper introduces NEUROQA, a new, massive "exam" designed specifically to stop robots from cheating and force them to actually look at the 3D brain images.

Here is how the paper breaks it down, using simple analogies:

1. The Problem: The "Text-Only" Cheat Code

Previous tests for medical AI were like giving a student a multiple-choice quiz where the questions were written in a way that gave away the answers.

The 2D Problem: Most old tests showed the AI just a flat, 2D slice of a brain (like a single page of a book). But real brain scans are 3D volumes (like a whole book). You can't understand the whole story by reading just one page.
The "Shortcut" Problem: The paper found that if you took the image away from these old tests, the AI still got 70–99% of the answers right! This means the AI wasn't "seeing" the brain; it was just guessing based on text patterns (e.g., "If the question mentions 'Parkinson's,' the answer is usually 'Yes'").

2. The Solution: NEUROQA (The "Honest" Exam)

The authors built a new benchmark called NEUROQA (56,953 questions from 12,977 real patients). Think of this as a rigorous, anti-cheating exam with three special rules:

The 3D Rule: Every question comes with the entire 3D brain volume, not just a flat slice. The AI has to navigate the brain in 3D space, just like a doctor does.
The "No-Image" Stress Test: To prove the AI is actually looking, they run a test where they hide the image. If the AI's score drops significantly when the image is gone, it proves the AI was actually using the image. If the score stays high, the AI is just guessing based on text.
The "Human Ceiling": They didn't just compare the AI to random guessing; they compared it to real human doctors. Two doctors (a radiology resident and a neurosurgery resident) took the same test using a 3D viewer. Their average score was about 49%. This sets a "human visual limit." If an AI scores lower than a human looking at the same view, it hasn't mastered the task yet.

3. The Two Types of Questions

The exam has two types of questions, which is a crucial distinction:

Image-Grounded (The "Look and See" Questions): These are things you can spot just by looking at the 3D scan, like "Is the left side of the brain smaller than the right?" or "What color is this spot on the scan?"
Image-Informed (The "Math and Context" Questions): These require knowing specific numbers that you can't see with your eyes. For example, "Is the volume of this brain part below the 5th percentile?" You can't eyeball a precise milliliter measurement. The "correct" answer comes from a computer calculation (FreeSurfer), not just human vision. This tests if the AI can extract precise data, not just guess.

4. The Results: The AI is Still Struggling

The authors tested the smartest AI models available today (like the latest versions of GPT, Gemini, and Claude) on this exam.

The Cheating Check: They cleaned up the questions so that if you removed the image, the AI would only get about 44.6% right (which is barely better than random guessing). This proves the exam is fair and doesn't give away answers.
The AI Performance: Even the best AI models only scored around 47.5% on the closed questions (Yes/No and Multiple Choice).
The Human Performance: The human doctors scored around 48.9%.
The Conclusion: Currently, the best AI models are not beating the human doctors, and in some cases, they are scoring lower than the text-only baseline. This means the AI is not yet reliably "seeing" the 3D brain better than a human can, nor is it extracting the precise quantitative data needed for a diagnosis.

5. How They Built It (The "Factory")

To make sure the exam was perfect, they didn't use AI to generate the questions (which would be circular logic). Instead, they built a deterministic pipeline.

Imagine a factory assembly line with 38 strict rules.
They took real patient data and ran it through this machine.
Human experts (doctors) reviewed the questions to make sure they made medical sense.
They stripped away any "clues" in the text that might let the AI guess without looking.
The result is a "gold standard" dataset where every answer is mathematically verified against the patient's actual scan data.

Summary

NEUROQA is a giant, honest test for medical AI. It forces the AI to look at full 3D brain scans and proves that, right now, even the smartest AI models are still struggling to understand these scans as well as a human doctor can. The paper doesn't claim the AI is ready for hospitals; instead, it provides a clear, measurable way to track when AI finally learns to "see" the brain properly.

Technical Summary: NEUROQA

Problem Statement

Current medical Visual Question Answering (VQA) benchmarks fail to adequately evaluate 3D brain MRI understanding. Existing resources are predominantly limited to 2D slices (e.g., VQA-RAD, PathVQA) or restricted to single clinical domains, failing to capture the inherently volumetric and context-dependent nature of clinical brain MRI reading. Furthermore, prior benchmarks suffer from "shortcut exploitation," where models achieve high accuracy by relying on text priors, cohort names, or answer-position biases rather than visual evidence. As demonstrated by Asadi et al., frontier Vision-Language Models (VLMs) often retain 70–99% of their accuracy when images are removed, indicating that benchmark scores may overstate true image understanding. Additionally, the absence of a human visual baseline makes it difficult to distinguish between modeling failures and the inherent perceptual limits of visual inspection.

Methodology

NEUROQA is a large-scale, image-grounded benchmark designed to address these gaps through a rigorous, deterministic construction pipeline.

Data Composition: The benchmark comprises 56,953 QA pairs across 12,977 subjects from 12 neuroimaging datasets, spanning ages 5–104 and five clinical domains (Alzheimer's, Parkinson's, tumors, white matter disease, and neurodevelopment).
3D Volumetric Grounding: Unlike prior 2D approaches, every item is paired with a full 3D volume in MNI-152 space.
Reasoning Categories: The 203 question templates are organized into 11 categories, split into:
- Image-grounded (131 templates): Answerable via direct visual inspection on a standard 3-plane viewer (e.g., Modality, Anatomy, Location, Signal).
- Image-informed (72 templates): Ground truth derived from quantitative volumetry (FreeSurfer) or clinical instruments (e.g., CDR, UPDRS), which cannot be recovered by visual inspection alone (e.g., Diagnosis, Severity, Reasoning).
Construction Pipeline: A 38-rule deterministic pipeline (seed=42) reduced an initial pool of 2.7M candidates to the final set without using Large Language Models (LLMs) for generation. The process included:
- Expert Review: Two rounds involving domain experts and clinicians to correct laterality conventions, signal wording, and diagnostic frameworks.
- Machine-Verified Audits: Seven rounds of automated checks for session alignment, age consistency, and image-path integrity.
- Shortcut Elimination: The final stage enforced answer balance (50/50 for Yes/No, uniform 25% for MCQ positions) and removed dominant templates. This reduced text-only accuracy from >80% to 44.6%, bringing it within 5.1 percentage points of the random chance floor (39.5%).
Release Architecture: A two-tier release system handles data governance:
- Tier 1 (Public): Direct release of 19,262 QA pairs from six permissively licensed datasets.
- Tier 2 (DUA): Deterministic generation scripts for 37,691 pairs from six datasets restricted by Data Use Agreements, allowing reproducible generation for researchers with access.
Evaluation Protocol: Models are evaluated on a held-out public test set (5,524 items) and a private test set (5,581 items). The protocol includes:
- Text-only Baseline: Measuring accuracy without image input to quantify shortcut reliance.
- Clinician Baseline: Two independent clinicians evaluated a 100-item frozen subset on a 3-plane NIfTI viewer.
- Image-Grounding Protocol: A three-stress-test framework (image-absent accuracy, fabrication rate, hallucination rate) to certify that performance gains stem from image understanding rather than text priors.

Key Contributions

Largest 3D Brain MRI Benchmark: NEUROQA is the largest publicly documented benchmark for 3D brain MRI QA, featuring 56,953 pairs, 12 datasets, and end-to-end preservation of volumetric structure.
Shortcut Auditing: Through template-level auditing and answer-distribution refinement, the benchmark successfully reduced text-only accuracy to near-random levels (44.6%), ensuring that residual accuracy is attributable to image content.
Clinician Visual Baselines: The study establishes a human visual reference, with clinicians achieving a mean closed-format accuracy of 48.9% on a frozen subset. This defines a perceptual ceiling for visual inspection alone.
Reproducible Two-Tier Release: The architecture supports open access while maintaining strict data governance for restricted datasets via deterministic generation pipelines.
Image-Grounding Protocol: A standardized method to verify that model performance is not driven by text priors, fabrication, or hallucination.

Results

Model Performance: On the test-public closed-format items (Yes/No + Multiple Choice), the best zero-shot VLM (Gemini-3.1-Pro) achieved 47.5% accuracy, and a supervised 3D CNN baseline achieved 43.7%.
Comparison to Baselines: Both the best VLM and the supervised baseline performed below the text-only majority-template floor of 49.4% and the clinician mean of 48.9%.
Shortcut Resistance: The text-only accuracy of 44.6% represents a shortcut margin of only +5.1 percentage points over random chance, the lowest among compared medical VQA benchmarks.
Clinician Analysis: Clinicians performed well on directly visible categories (e.g., Modality at 100%) but plateaued on image-informed categories (e.g., Reasoning at ~28.6%), confirming that ground truth for these categories is not visually recoverable.
Inter-rater Agreement: Clinician agreement was fair-to-moderate (Cohen's $\kappa$ = 0.40), with significant divergence in categories requiring multi-modal workup or precise spatial reasoning.

Significance and Claims

The paper claims that NEUROQA establishes a rigorous standard for evaluating 3D brain MRI understanding by explicitly separating image-driven performance from text-prior reliance. By demonstrating that current state-of-the-art VLMs and supervised 3D CNNs fail to exceed the text-only floor or the clinician visual baseline, the benchmark highlights a significant gap between current AI capabilities and the requirements for clinically grounded 3D reasoning.

The authors assert that NEUROQA is not intended to support claims about clinical deployment, prognosis, or treatment planning, but rather to provide a transparent evaluation culture where claims of neuroimaging capability must survive shortcut, fabrication, and hallucination checks. The benchmark is designed to be solvable in principle by models that can extract quantitative information from 3D volumes beyond what is visible to the human eye, particularly in image-informed categories.