RADAR: A Multimodal Benchmark for 3D Image-Based Radiology Report Review

The paper introduces RADAR, a multimodal benchmark comprising expert-annotated 3D abdominal CT scans and radiology report edits that enables the systematic evaluation of AI models on fine-grained clinical reasoning tasks, specifically image-text alignment and discrepancy assessment during the radiology report review process.

The Gist

Imagine you are a student pilot flying a plane for the first time. You write down your observations of the flight in a logbook: "The sky is clear, the engine sounds fine, and we are at 10,000 feet."

Later, your instructor, a seasoned captain, looks at your logbook, checks the actual flight data (the instruments, the weather radar, the engine sensors), and makes some changes. Maybe they cross out "engine sounds fine" and write "Engine vibration detected," or they add a note about a cloud formation you missed.

RADAR is a new tool designed to teach computers how to be that "super-smart instructor." Its job is to look at the student's report, look at the actual flight data (the medical images), and decide:

1. Did the instructor make the right change?
2. How dangerous was the mistake the student made?
3. What kind of change did the instructor make (fixing a lie, adding a missing detail, or just clarifying something vague)?

The Problem: Why Do We Need This?

In the real world, radiologists (doctors who read X-rays and CT scans) often work in shifts. A trainee (the student) writes a "preliminary" report on a patient's scan. Later, a senior doctor (the attending) reviews it and might change the diagnosis.

Sometimes these changes are tiny and harmless. Other times, the trainee missed a tumor or a broken bone, and the senior doctor catches it. If a computer could spot these differences before the senior doctor even looks, it could save lives.

But here's the catch: Current AI is bad at this.
Most AI models today are like spell-checkers. They can tell if a sentence is grammatically wrong or if a word doesn't make sense. But they can't look at a 3D CT scan (which is like a giant, complex block of jelly with layers inside) and say, "Hey, this report says the liver is healthy, but the scan clearly shows a dark spot here."

The Solution: The RADAR Benchmark

The researchers created RADAR (Radiology Report Review). Think of it as a giant, tricky exam for AI models.

• The Test Material: Instead of fake errors made up by a computer, they used real medical cases from a hospital. They took actual CT scans of abdomens, the trainee's original report, and the senior doctor's final corrections.
• The Challenge: The AI is given the scan and the trainee's report. It is then shown a "suggested edit" (a change the AI thinks should be made). The AI has to answer three questions:
  1. Agreement: Is this edit actually supported by the picture? (Yes, No, or Sort of?)
  2. Severity: If this edit is ignored, how bad is it? (Critical, Moderate, or Negligible?)
  3. Type: What kind of edit is it? (A correction, an addition, or just a clarification?)

The Analogy: The "Three-Legged Stool"

To pass the RADAR test, an AI model needs to sit on a three-legged stool. If one leg is missing, it falls over.

1. Leg 1 (Vision): It must understand the 3D image. It can't just read the words; it has to "see" the anatomy.
2. Leg 2 (Reasoning): It must connect the image to the text. "The text says 'no fracture,' but the image shows a crack."
3. Leg 3 (Judgment): It must understand medical stakes. "Missing a broken bone is bad, but missing a tiny shadow might be okay."

What Happened When They Tested It?

The researchers tested several of the world's most powerful AI models (like Google's Gemini and Alibaba's Qwen) on this exam.

• The Good News: The AI models were great at spotting linguistic patterns. They could easily tell if a sentence was a "correction" or an "addition."
• The Bad News: They struggled with the hard stuff.
  • The "Hallucination" Trap: When the researchers tricked the AI with fake edits that didn't match the image, the AI often got confused and agreed with them anyway.
  • The "Severity" Gap: The AI had a hard time deciding if a mistake was "critical" or just "minor." It's like a student who knows the answer is wrong but can't tell if it's a spelling error or a math error that causes a plane crash.
  • More Data ≠ Better: Interestingly, feeding the AI more slices of the CT scan (more data) didn't always make it smarter. Sometimes, looking at too many images at once confused the model, just like trying to read 50 books at once might make you miss the plot of one.

Why Does This Matter?

RADAR isn't just a test; it's a safety net.

Imagine an emergency room at 3 AM. A trainee radiologist is tired and writes a report. A senior doctor isn't available for another hour. If an AI system trained on RADAR could step in and say, "Wait, the report says 'normal,' but the scan shows a bleed. This is a CRITICAL discrepancy," it could alert the team immediately.

This paper shows that while AI is getting good at reading words, it still has a long way to go before it can truly "see" and "judge" medical images like a human doctor. RADAR gives researchers a clear map of where the AI is failing, so they can build better, safer tools for the future.

Technical Summary

Here is a detailed technical summary of the paper "RADAR: A Multimodal Benchmark for 3D Image-Based Radiology Report Review."

1. Problem Statement

Radiology reports for the same patient examination often contain clinically meaningful discrepancies due to interpretation differences, reporting variability, or evolving assessments. These discrepancies occur in approximately 3–5% of routine interpretations and can have significant downstream impacts on patient care, particularly in emergency settings where preliminary reports guide immediate decisions before final review.

Current Limitations:

• Lack of Standardized Benchmarks: Existing datasets for radiology error analysis are limited, especially for volumetric 3D CT scans.
• Synthetic vs. Real Errors: Prior work often relies on synthetically introduced text errors (e.g., word insertion/deletion) or text-only detection, which fails to capture clinically meaningful, image-evidenced discrepancies.
• Workflow Mismatch: Existing Vision-Language Models (VLMs) are typically trained for report generation or Visual Question Answering (VQA), not for the specific task of reviewing a preliminary report against an image to validate a candidate edit.
• Evaluation Gap: There is no standardized framework to evaluate whether a proposed edit is supported by imaging evidence, its clinical severity, or its specific type (correction, addition, clarification).

2. Methodology: The RADAR Benchmark

RADAR is a multimodal benchmark designed to simulate the real-world clinical workflow where a trainee radiologist's preliminary report is reviewed and revised by an attending radiologist.

A. Dataset Construction

• Source: Abdominal and pelvic CT examinations from the Emergency Department (ED) at Harborview Medical Center (UW Medicine).
• Composition: 50 total cases (20 validation, 30 test), including both daytime and overnight shifts.
  • Daytime: Comprehensive preliminary reports refined by attendings.
  • Overnight: "Quick-look" preliminary reports focusing on acute findings, later expanded/corrected.
• Data Structure: Each entry consists of:
  1. Volumetric CT Images: Multiple DICOM series.
  2. Preliminary Report: Authored by residents/fellows.
  3. Candidate Edit: A specific text modification suggested to resolve a discrepancy.
• Annotation Process:
  • Discrepancies were identified using GPT-4o and verified by an attending radiologist.
  • Gold-Standard Labels: A board-certified radiologist (10+ years experience) annotated each candidate edit on three dimensions:
    1. Agreement: Does the edit align with imaging evidence? (Agree, Partially Agree, Disagree).
    2. Clinical Severity: What is the potential impact? (Critical, Moderate, Negligible).
    3. Edit Type: What is the nature of the change? (Correction, Addition, Clarification).
• Balancing: To address class imbalance (natural data had few "Disagree" cases), the authors introduced Synthetic Disagree edits. These are plausible but incorrect modifications generated by GPT-5.2 and verified by experts to ensure models are tested on unsupported edits.

B. Task Definition

The task requires a multimodal model to evaluate a candidate edit given the CT volume and preliminary report, without access to the final attending report. The model must output:

1. Image-grounded Agreement: (Agree/Partial/Disagree).
2. Clinical Severity: (Critical/Moderate/Negligible).
3. Edit Type: (Correction/Addition/Clarification).

C. Baseline Systems & Input Strategies

The authors evaluated several state-of-the-art Vision-Language Models (VLMs) under different volumetric input settings:

• Models: Gemini-2.5-Pro, Gemini-3-Pro, and Qwen3.5-plus.
• Input Strategies:
  • Series Selection: An auxiliary model (GPT-OSS-20B) selects the most relevant CT series based on metadata and edit text.
  • Slice Sampling: Uniformly sampling 10, 20, or 50 slices.
  • Video Input: Treating the entire series as a video sequence.

D. Evaluation Metrics

• Accuracy: Reported separately for Agreement, Severity, and Edit Type.
• Composite Score: A strict metric requiring an exact match on all three dimensions (Agreement + Severity + Edit Type) to award a point. This measures end-to-end reasoning capability.

3. Key Contributions

1. First Real-World Multimodal Benchmark: RADAR is the first benchmark specifically designed for image-grounded radiology report discrepancy analysis using real trainee-to-attending revisions rather than synthetic errors.
2. Fine-Grained Evaluation Framework: Moves beyond binary error detection to jointly assess agreement, clinical severity, and edit taxonomy.
3. Empirical Baselines: Provides a comprehensive evaluation of leading foundation models (Gemini, Qwen) across various 3D input modalities (slice sampling vs. video), establishing a baseline for future research.
4. Clinical Grounding: The dataset reflects actual clinical workflows, including the distinction between daytime and overnight reporting pressures.

4. Results

• Performance Overview:
  • Edit Type Classification: Models performed well (0.78–0.84 accuracy), suggesting they can easily identify the semantic function of an edit.
  • Agreement & Severity: Performance was moderate (0.46–0.70 for Agreement; 0.46–0.56 for Severity), indicating significant difficulty in cross-modal alignment and clinical judgment.
  • Composite Score: Generally low (0.16–0.34 on Mixed set; up to 0.399 on Natural set), highlighting the difficulty of simultaneously satisfying all three reasoning criteria.
• Input Strategy Findings:
  • Slice Count: Increasing the number of slices (from 10 to 50) did not consistently improve performance. For example, Gemini-3-Pro achieved its best Composite Score with 50 slices on the Natural set, but Qwen3.5-plus performed best with 20 slices.
  • Video vs. Slices: Video-style input (processing all slices as a sequence) did not reliably outperform slice-based sampling.
• Model Comparison: Performance differences were primarily driven by the model family (e.g., Gemini-3-Pro generally outperformed Gemini-2.5-Pro) rather than the input configuration alone.

5. Significance and Future Work

• Clinical Safety: RADAR demonstrates that current VLMs struggle with "clinically nuanced reasoning." A discrepancy-aware model could serve as a critical verification layer in AI-assisted pipelines, flagging edits that lack image support or carry high clinical risk before they influence patient care.
• Workflow Integration: The benchmark is particularly relevant for Emergency Departments and resource-limited settings where attending review may be delayed.
• Future Directions: The authors plan to expand the benchmark to include additional modalities (e.g., MRI), other anatomical regions, and longitudinal reasoning (comparing current scans with prior exams) to further enhance safety and robustness.

In summary, RADAR fills a critical gap in medical AI by shifting the focus from generating reports to critically reviewing them against 3D imaging evidence, providing a rigorous testbed for developing safer, more reliable clinical decision support systems.