💬 NLP

ReportQA: QA-Based Radiology Report Evaluation

The paper introduces ReportQA, a novel radiology report evaluation framework that leverages large language models to generate and answer structured clinical questions, resulting in the QAScore metric which demonstrates superior alignment with radiologist judgments compared to existing evaluation methods.

Original authors: Yiming Shi, Shaoshuai Yang, Xi Chen, Haolin Li, Hengyu Zhang, Che Jiang, Kaiwen Wang, Xun Zhu, Dong Xie, Fei Wang, Dejing Dou, Miao Li, Ji Wu

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yiming Shi, Shaoshuai Yang, Xi Chen, Haolin Li, Hengyu Zhang, Che Jiang, Kaiwen Wang, Xun Zhu, Dong Xie, Fei Wang, Dejing Dou, Miao Li, Ji Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading a student's essay about a medical scan. In the past, teachers (or computers) used two main ways to grade these essays:

The "Word Count" Method: They checked how many words matched the "perfect" essay. If the student used the same fancy words, they got a high score. But this is like grading a recipe just because it uses the same ingredients as a famous chef's, even if the final dish tastes terrible. It doesn't tell you if the medical advice is actually correct.
The "Checklist" Method: They looked for a few specific things, like "Is there a tumor?" or "Is there fluid?" If the student mentioned these, they got points. But this is like checking a car only for headlights and ignoring the engine, brakes, or tires. It misses the tiny, crucial details that doctors actually care about, like where exactly the tumor is or how big it is.

The Problem:
Current computer programs that write radiology reports are stuck in this loop. They are good at sounding like doctors, but they often miss the fine-grained details or make up things that aren't there. We need a better way to test them that feels like how a real doctor actually uses a report.

The Solution: ReportQA
The authors of this paper, from Tsinghua University, built a new testing system called ReportQA. Think of it as turning the report from a "story to be read" into a "quiz to be taken."

Here is how it works, using a simple analogy:

1. The "Knowledge Tree" (The Syllabus)

First, the researchers asked real radiologists to draw a giant "family tree" of medical knowledge.

The Trunk: Major body parts (like the Chest or Brain).
The Branches: Specific conditions (like Pneumonia or a Fracture).
The Leaves: Tiny details (Is it on the left or right? Is it sharp or blurry? Is it new or old?).
This tree acts as the syllabus. It ensures the computer knows exactly what details matter, down to the smallest leaf.

2. The "Translator" (Structuring the Report)

Radiology reports are usually written in messy, free-flowing paragraphs. The team used a powerful AI (a Large Language Model) to act as a translator. It takes the messy paragraph and forces it into a neat, structured format based on the "Knowledge Tree" syllabus. It's like taking a rambling story and turning it into a structured spreadsheet where every fact has its own specific box.

3. The "Quiz Generator" (Creating Questions)

Once the report is structured, the system automatically generates hundreds of multiple-choice questions based on that data.

Example Question: "Is there a 'patchy' (shape) 'pneumonia' (disease) in the 'right lung' (location)?"
The Trick: They also create "Negative Questions" (e.g., "Is there a broken bone?") to make sure the computer isn't just guessing "Yes" to everything.
Quality Control: Before the test starts, the system filters out any questions that are too easy or don't actually need the report to answer. This ensures the test is fair and challenging.

4. The "Judge" (Taking the Test)

Now, the real test begins. The computer system that generated the report is given the report as "context" (like a student looking at their own essay). A separate, very smart AI acts as the "Judge." The Judge reads the report and tries to answer the hundreds of generated questions.

If the report says "No pneumonia," but the Judge answers "Yes, there is pneumonia," the report gets a point deducted.
If the report misses a detail about the size of a tumor, the Judge gets that question wrong, and the report loses points.

The Result: QAScore

Instead of a simple grade, they created a new metric called QAScore. It's a single number that tells you how well the report holds up under a rigorous quiz.

The Finding: When they tested this new system, they found that the old "Word Count" and "Checklist" methods were terrible at catching errors. QAScore, however, matched what human radiologists thought was right or wrong much better.

What Did They Learn?

The paper also used this new test to peek under the hood of current medical AI models and found two surprising things:

The "Negative Bias": Current models are afraid to say "Yes." If they aren't 100% sure, they tend to say "Nothing is wrong" (a negative answer) just to be safe. It's like a student who, when unsure of the answer, just writes "I don't know" instead of taking a guess.
The "Fine-Grained" Struggle: These models are okay at spotting big things (like "Is there a tumor?"), but they are terrible at the tiny details (like "Is the tumor on the left or right side?"). They seem to memorize the style of a report but don't truly understand the visual evidence inside the scan.

The Bottom Line:
The authors suggest that instead of asking AI to "write a whole report" (which is hard and prone to errors), we should ask it to "answer specific questions" about the scan. This "Question-Driven" approach seems to work much better, forcing the AI to look at the actual evidence rather than just guessing based on what it thinks a report should sound like.

They have released all their "Knowledge Trees," the "Quiz Questions," and the code so other researchers can use this new, fairer way to grade medical AI.

Technical Summary: ReportQA: QA-Based Radiology Report Evaluation

Problem Statement

The advancement of automated radiology report generation (RRG) is currently constrained by the lack of reliable evaluation frameworks. Existing Radiology Report Evaluation (RRE) methods suffer from two primary limitations:

Natural Language Generation (NLG) Metrics: Metrics like BLEU and ROUGE rely on surface-level lexical overlap, which fails to capture clinical correctness. A report may be lexically similar to a reference but clinically inaccurate regarding key entities and attributes.
Clinical Efficacy (CE) Metrics: While metrics like CheXbert and RadGraph evaluate the presence of specific clinical entities, they are limited to coarse-grained existence modeling. They often fail to capture fine-grained attributes (e.g., location, shape, size, certainty) and rely heavily on supervised models trained with extensive manual annotations, making them difficult to extend to new entities or attributes.

Furthermore, current RRG systems often struggle to learn fine-grained clinical representations and exhibit strong negative prior biases, a phenomenon not adequately detected by existing report-level metrics.

Methodology

The authors propose ReportQA, a flexible, clinical-related evaluation framework that treats radiology reports as context for Question Answering (QA). The framework operates through a four-stage pipeline:

1. Dataset Preparation

The framework aggregates diverse radiology report datasets covering multiple imaging modalities (X-ray, CT, MRI, Ultrasound) and anatomical regions (Brain, Chest, Abdomen, Knee). Sources include MIMIC-CXR, CTRG-Brain, CT-RATE, and AMOS-MM.

2. Structured Report Extraction

To align with real-world clinical scenarios, free-form reports are transformed into structured formats using radiologist-defined knowledge trees.

Knowledge Trees: Radiologists construct trees for clinical entities (findings and diagnoses) and a comprehensive set of 15 clinical attributes (presence, location, distribution, number, dimension, density, shape, margin, enhancement, internal features, secondary effects, severity, chronicity, clinical score, and certainty).
Extraction Process: Large Language Models (LLMs) extract structured information from raw reports in a two-stage process: preliminary extraction followed by refinement using the knowledge trees to ensure alignment with predefined nodes and eliminate invalid entities.

3. QA Generation

Structured reports are converted into QA pairs using predefined templates. The generation includes three categories:

Base (Leaf-level): Questions targeting specific (entity, attribute) combinations.
Hierarchical (Ancestor-level): Questions targeting coarser-grained concepts to account for partial correctness (e.g., "brain atrophy" vs. "diffuse brain atrophy").
Negative: Questions regarding entities absent in the ground-truth report to test for false positives.
Quality Control: A two-stage filtering process using LLMs removes QA pairs that can be answered without context or have low accuracy when the ground-truth report is provided as context. This results in approximately 660,000 high-quality QA pairs.

4. QA-Based Evaluation and QAScore

During evaluation, a generated report serves as the context for an LLM judge model to answer the filtered QA pairs. Performance is quantified using QAScore, an aggregated metric defined as the harmonic mean of two components:

$Score_{pos}$ : Measures positive consistency (accuracy on base and hierarchical questions). If the presence of an entity is missed, all associated attribute questions for that entity are marked incorrect.
$Score_{neg}$ : Measures the avoidance of spurious entities, calculated as $exp(-\lambda \cdot FPR_{neg})$ , where $FPR_{neg}$ is the false positive rate for negative questions.

Key Contributions

ReportQA Framework: A novel, QA-based RRE framework that supports detailed quantitative analysis of RRG systems by simulating how clinicians interact with reports (using them as context for diagnostic queries).
QAScore Metric: A new metric that demonstrates stronger alignment with radiologist judgments compared to existing NLG and CE metrics.
Discovery of Failure Modes: The framework reveals that current report-based inference paradigms struggle with fine-grained clinical attributes and exhibit strong negative prior biases (tending to classify entities as absent even without image evidence).
Resource Release: The authors release the knowledge trees, structured reports, QA pairs, and pipeline code to ensure reproducibility and extensibility.

Experimental Results

The authors conducted comprehensive experiments on state-of-the-art Vision-Language Models (VLMs), including proprietary APIs (GPT-5.4, Gemini, Claude) and open-source models (Qwen, InternVL, Hulu-Med, CT-CHAT).

Alignment with Radiologists: On the RadEvalX dataset, QAScore achieved the highest correlation with human-annotated errors (Pearson = 0.4507, Spearman = 0.4612), outperforming metrics like GREEN, RaTEScore, CheXbert, and BLEU.
Model Performance: Proprietary VLMs generally outperformed open-source models. Performance was higher on 2D chest X-rays (MIMIC-CXR) than on 3D CT datasets, highlighting the challenge of 3D medical image understanding.
Attribute Difficulty: Models performed well on "presence" judgments but struggled significantly with fine-grained attributes such as margin, chronicity, and enhancement.
Inference Paradigms:
- Report-based vs. Question-driven: Report-based supervised fine-tuning (SFT) showed limited gains in fine-grained entity perception compared to untrained question-driven inference.
- Bias Analysis: Report-based inference exhibited strong negative prior biases (classifying most entities as absent without image input). In contrast, question-driven inference showed a milder positive bias.
- Visual Grounding: Even without image input, VLMs achieved non-trivial scores based on textual priors, suggesting a reliance on statistical patterns rather than visual evidence. However, question-driven inference with images consistently outperformed image-free inference.

Significance and Claims

The paper claims that ReportQA provides a more faithful measure of report utility by aligning evaluation with the clinical use of radiology reports as information transfer media. The authors posit that question-driven inference represents a more promising alternative to traditional report-based inference for RRG systems, as its biases can be mitigated through mixed positive and negative training samples. The framework highlights that current models have not yet mastered the alignment of visual evidence with structured text generation, particularly for fine-grained clinical attributes, identifying this as a critical direction for future research.