🤖 AI

VERT: Reliable LLM Judges for Radiology Report Evaluation

This paper introduces VERT, a reliable LLM-based metric for evaluating radiology reports across diverse modalities and anatomies, demonstrating through extensive correlation analysis and fine-tuning that it significantly outperforms existing methods in alignment with expert judgments while offering substantial efficiency gains.

Original authors: Federica Bologna, Jean-Philippe Corbeil, Matthew Wilkens, Asma Ben Abacha

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Federica Bologna, Jean-Philippe Corbeil, Matthew Wilkens, Asma Ben Abacha

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading a stack of student essays. Your job is to check if the students correctly described a medical scan (like an X-ray or MRI) and wrote a report about what they saw.

In the past, computers tried to grade these reports using rigid rules, like counting how many words matched between the student's essay and the teacher's answer key. But medical reports are tricky. A computer might miss that a student wrote "a small shadow" instead of "a tiny spot," even though they mean the same thing. Or, the computer might get confused if the report is about a knee instead of a chest, because it was only trained on chest X-rays.

This paper introduces a new, smarter way to grade these reports using Large Language Models (LLMs)—the same kind of AI that powers chatbots. The authors call their new system VERT.

Here is a breakdown of what they did, using some everyday analogies:

1. The Problem: The "One-Size-Fits-All" Grader

Previous AI graders were like a teacher who only taught Chest X-rays. If you handed them a report about a broken leg or a brain scan, they would get confused and give bad grades. They were too specialized and couldn't handle the variety of medical imaging.

2. The Solution: The "Super-Grader" (VERT)

The authors created VERT. Think of VERT as a highly experienced, all-around medical professor who has seen every type of scan imaginable.

How it works: Instead of just counting matching words, VERT reads the report like a human doctor. It looks for the meaning. Did the student miss a critical finding? Did they invent a problem that wasn't there?
The Result: VERT is much better at agreeing with human experts than previous AI tools. It improved the grading accuracy by up to 11.7% compared to the next best method.

3. The Experiment: Testing Different "Brains"

The researchers didn't just use one AI; they tested many.

The "Big Brains" vs. "Small Brains": They tested massive, expensive AI models (like GPT-4) and smaller, cheaper ones.
The "Thinking" Test: They asked some AIs to "think step-by-step" before grading (like a student showing their work). Surprisingly, for this specific task, thinking too hard didn't always help. Sometimes, the smaller, faster models actually did a better job than the massive ones.
The "Cheating" Test (Few-Shot): They tried giving the AI a few examples of good and bad reports before asking it to grade a new one. This helped a little bit, but it wasn't a magic bullet.

4. The Secret Sauce: "Fine-Tuning" (Teaching the AI)

This is the most exciting part. The researchers took a powerful but generic AI model (Qwen3) and gave it a crash course using just 1,300 examples of graded reports.

The Analogy: Imagine taking a brilliant but inexperienced medical student and giving them a summer internship where they review 1,300 reports with a senior doctor. By the end, they become an expert.
The Result: This "fine-tuned" model became 25% more accurate than the untrained version. Even better, it was 37 times faster and much cheaper to run than the giant, expensive AI models. It's like upgrading from a slow, fuel-guzzling truck to a high-speed electric sports car.

5. The Reality Check: Where They Still Struggle

The authors also played "gotcha" with the AI. They secretly inserted fake errors into reports to see if the AI would catch them.

What they found: The AI is great at spotting obvious mistakes (like saying a patient has a broken leg when they don't).
Where it fails: It sometimes misses subtle errors, like getting the location of a problem wrong (saying "left knee" instead of "right knee") or missing a comparison to a previous scan. It's like a student who gets the main idea right but misses the tiny details.

The Big Takeaway

This paper proves that we don't need massive, expensive super-computers to grade medical reports accurately. By using a smart prompting strategy (VERT) and giving a standard AI a little bit of specialized training (fine-tuning), we can create a reliable, fast, and cheap "AI Teaching Assistant" that helps doctors ensure their reports are accurate.

In short: They built a better grading system for medical reports that is smarter, faster, and cheaper than anything we've had before, but they also admitted it still needs a human to double-check the tiny details.

1. Problem Statement

Current literature on radiology report evaluation is heavily skewed toward chest X-rays and relies on two main approaches:

Fine-tuning small models on specific datasets (e.g., CheXbert, RadGraph).
LLM-based metrics designed primarily for error detection (e.g., RadFact, GREEN).

Key Limitations:

Lack of Generalizability: Models optimized for chest X-rays often fail when applied to other modalities (CT, MRI, Ultrasound) or anatomical regions due to the inherent variability in medical imaging conventions.
Uncertainty in Configuration: It is unclear which Large Language Models (LLMs) (open vs. closed source, reasoning vs. non-reasoning) or prompting strategies (zero-shot, few-shot, formula-based) serve as the most reliable "judges" for radiology reports.
Evaluation Gap: There is a need for a robust metric that correlates strongly with expert radiologist judgments across diverse clinical scenarios, not just chest radiographs.

2. Methodology

The authors conducted a comprehensive study using two expert-annotated datasets:

RadEval: 148 chest X-ray studies with detailed error annotations (6 error categories + grammar).
RaTE-Eval: A paragraph-level benchmark spanning 9 modalities and 22 anatomies with expert scores (0–5).

The study evaluated several dimensions:

A. Prompting Strategies & Metric Design

The authors compared four zero-shot prompting formulations using GPT-4.1-mini and Gemini-2.5-flash:

GREEN: Chain-of-thought prompting to identify errors and compute a score post-hoc.
VERT (Proposed): Direct prompting for the model to output an overall accuracy score (0.0–1.0) based on the error analysis, bypassing intermediate formulaic calculations.
Formula-based: Instructing the model to compute a score using a weighted equation ( $Score = \frac{Matched}{Matched + 2 \times SigErrors + 0.5 \times InsignifErrors}$ ).
Rubric-based: Using a discrete scoring rubric instead of a formula.

B. Model Comparison

The study tested a wide range of models:

Closed-source: GPT-4.1 family (Nano, Mini, Standard), GPT-5, Gemini 3 Flash, Claude Opus/Sonnet 4.6.
Open-source: Qwen3 (30B and 4B), including "thinking" (reasoning) variants.
Reasoning Analysis: Investigated whether "thinking" (extended reasoning traces) improves correlation with human judgments.

C. Advanced Techniques

Ensembling: Averaging scores and Linear Regression ensembles combining outputs from GPT-4.1, Claude Opus, and Gemini 3 Flash.
Few-Shot Prompting: Tested random shots vs. curated shots containing specific error types (e.g., false predictions, omissions).
Parameter-Efficient Fine-Tuning (PEFT): Fine-tuned Qwen3-30B-A3B using LoRA (Low-Rank Adaptation) on the RaTE-Eval training set (1,300 samples).

D. Error Detection & Categorization

To assess granular alignment, the authors performed a systematic error injection study:

Injected clinically significant errors (False Predictions and Missing Findings) into reference reports.
Evaluated the models' ability to detect and categorize these specific errors against human annotations.

3. Key Contributions

Introduction of VERT: A novel LLM-based metric that prompts the model to directly generate an overall accuracy score. It outperforms existing methods (RadFact, GREEN, FineRadScore) across multiple modalities.
Comprehensive Benchmarking: The first large-scale comparison of closed/open-source models and prompting strategies specifically for multi-modality radiology evaluation.
Demonstration of Lightweight Adaptation: Proved that LoRA fine-tuning on a small dataset (1,300 samples) significantly boosts performance and inference speed compared to zero-shot prompting.
Error Analysis: A systematic study revealing that while LLMs correlate well with overall scores, they struggle to accurately count specific error types (particularly omissions) and often over/underestimate error counts depending on the baseline.

4. Key Results

Correlation with Expert Judgments

VERT Superiority: VERT achieved the highest correlation with human annotations. On RaTE-Eval, it improved correlation by up to 11.7% relative to GREEN.
Best Models: Claude 4.6 (Opus/Sonnet) and GPT-4.1 families generally outperformed others. Notably, "thinking" (extended reasoning) did not consistently improve correlation; in some cases (GPT-5, Claude), it decreased performance.
Prompting Impact: Direct scoring (VERT) outperformed formula-based and rubric-based prompts. Providing formulas or rubrics often degraded performance, likely due to the model's difficulty in adhering to rigid mathematical constraints during generation.

Ensembling and Fine-Tuning

Ensembling: Linear regression ensembles of multiple models (GPT, Claude, Gemini) improved correlation by 22.7% over the best single model baseline.
Fine-Tuning: Fine-tuning Qwen3-30B on 1,300 RaTE-Eval samples yielded a 25% gain in correlation over zero-shot performance.
Efficiency: The fine-tuned Qwen3 model reduced inference time by 37.2x (36 seconds vs. 22.3 minutes for Claude Opus via API) while maintaining high accuracy.

Error Detection Capabilities

Counting Errors: Models tend to overestimate errors when the true count is low (<6) and **underestimate** when high (>6).
Categorization:
- Models performed well on False Predictions (F1 ~0.90–0.92 on RaTE-Eval).
- Models struggled significantly with Missing Findings/Omissions (F1 ~0.69–0.72), often failing to detect when a critical finding was omitted.
- Performance dropped sharply for complex error types (anatomical location, severity, comparisons).

5. Significance and Conclusion

This paper establishes that LLM-based judges are viable and effective for radiology report evaluation across diverse modalities, moving beyond the limitations of chest-X-ray-specific models.

Reliability: The proposed VERT metric offers a robust, generalizable approach for evaluating radiology reports.
Efficiency: The study demonstrates that lightweight adaptation (LoRA) is a highly effective strategy, offering superior performance and massive speed/cost reductions compared to relying solely on massive closed-source APIs.
Future Directions: While overall scoring is reliable, the inability to accurately count specific error types (especially omissions) highlights a need for future work focused on granular error detection and evaluation methods that do not require reference reports.

In summary, VERT combined with parameter-efficient fine-tuning represents a state-of-the-art, cost-effective solution for automated radiology report evaluation in clinical and research settings.