💬 NLP

ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation

The paper introduces ReXrank, a public leaderboard and standardized evaluation framework featuring the large-scale ReXGradient dataset and multiple metrics to objectively assess and compare AI models for automated chest X-ray report generation.

Original authors: Xiaoman Zhang, Hong-Yu Zhou, Xiaoli Yang, Oishi Banerjee, Julián N. Acosta, Mohammed Baharoon, Josh Miller, Ouwen Huang, Pranav Rajpurkar

Published 2026-08-13

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiaoman Zhang, Hong-Yu Zhou, Xiaoli Yang, Oishi Banerjee, Julián N. Acosta, Mohammed Baharoon, Josh Miller, Ouwen Huang, Pranav Rajpurkar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where a computer can look at a blurry black-and-white picture of your ribs and instantly write a doctor's report about what's wrong. This isn't magic; it's a branch of science called artificial intelligence (AI) applied to medicine, specifically radiology. For years, scientists have been teaching computers to "read" X-rays, but there's been a massive problem: everyone was playing by different rules. Some teams tested their AI on one set of pictures, others on a different set, and they used different ways to grade how good the AI's writing was. It was like having a thousand different schools grading math homework with different answer keys—you couldn't tell who was actually the best student. This paper steps into that chaotic classroom to build a single, fair, and giant scoreboard where every AI can be tested under the same conditions.

The paper introduces ReXrank, a public leaderboard designed to be the ultimate referee for AI that writes chest X-ray reports. Think of it as a high-stakes cooking competition, but instead of chefs, the contestants are computer models, and instead of judging a soufflé, they are judging how well a robot can describe what it sees in a medical image. The authors gathered a massive, secret "test kitchen" called ReXGradient, containing 10,000 real chest X-ray studies from 67 different hospitals across the United States. This is the "final exam" that no AI has seen before. They also included three other public datasets to make sure the models aren't just memorizing answers but actually learning to cook.

To grade the robots, the researchers didn't just use one ruler; they used eight different ones. Some rulers check if the words sound like a human wrote them (like checking if a sentence flows well), while others are specialized medical rulers that check if the AI correctly identified specific diseases or didn't accidentally say a patient has a broken bone when they don't. They even used a "clinical error" ruler that acts like a strict editor, counting how many mistakes the AI made that a real doctor would catch.

When the dust settled, the results were clear. One model, named MedVersa, stood out as the champion, consistently scoring the highest across almost all the tests, especially on the tough, secret ReXGradient dataset. It beat out even the famous general-purpose AI models like GPT-4V, which are good at many things but not necessarily trained specifically for medical reports. The paper suggests that models trained on a mix of different data sources tend to be more robust, while others struggled when the data looked different from what they were used to. Interestingly, the study found that some public datasets were too easy, acting like a practice test that didn't prepare the models for the real thing, whereas the private ReXGradient dataset was the true stress test that revealed which models were truly ready for the hospital.

The authors are careful to note that while MedVersa is currently the top performer, this isn't a "solved" problem. Medicine is complex, and the paper emphasizes that these models are still being evaluated for their ability to generalize to new, unseen situations. The goal of ReXrank isn't to declare a permanent winner, but to provide a standardized way for the scientific community to track progress, ensuring that when AI tools eventually help doctors, they are reliable, accurate, and safe for patients everywhere.

Technical Summary: ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation

Problem Statement

While AI-driven models have demonstrated significant potential in automating the generation of radiology reports for chest X-rays, the field lacks a standardized benchmark for objectively evaluating and comparing their performance. Existing datasets, such as MIMIC-CXR, suffer from inconsistent data splits, a lack of standardized evaluation metrics, and data distributions that fail to adequately test a model's ability to generalize to unseen, diverse clinical settings. This absence of a unified framework hinders reliable comparative analysis across different model architectures and limits the assessment of clinical robustness.

Methodology

To address these gaps, the authors introduce ReXrank, a public leaderboard and challenge designed to assess AI-powered radiology report generation. The methodology is built upon a comprehensive evaluation framework involving four distinct datasets and eight diverse metrics.

Datasets

ReXrank evaluates models on a combination of public and private data to ensure robustness across varying distributions:

ReXGradient: A large-scale private test dataset comprising 10,000 studies from 67 U.S. medical sites and 7,004 patients. It serves as the primary benchmark for assessing generalization to diverse, real-world clinical environments.
MIMIC-CXR: The official test split (2,347 studies) from this large public dataset.
IU-Xray: The official test split (590 studies) from this public dataset.
CheXpert Plus: The validation set (200 studies), used as a test split since no official test set is available.

The framework evaluates models on two specific tasks: generating the Findings section only, and generating both Findings and Impressions sections.

Evaluation Metrics

The framework employs eight metrics to provide a multi-dimensional view of model performance, ranging from lexical similarity to clinical accuracy:

BLEU-2: Measures n-gram precision.
BERTScore: Computes cosine similarity between BERT embeddings of generated and ground-truth reports.
SembScore: Calculates cosine similarity between indicator vectors of 14 pathologies extracted by CheXbert.
RadGraph-F1: Evaluates the overlap in clinical entities and relations extracted by RadGraph.
RadCliQ-v1: A composite metric combining BLEU, BERTScore, SembScore, and RadGraph-F1. It is used as the primary ranking metric (reported as its reciprocal, $1/\text{RadCliQ-v1}$ , so higher is better).
RaTEScore: An entity-aware metric robust against medical synonyms and sensitive to negation.
GREEN: An LLM-based metric that identifies and explains clinically significant errors.
FineRadScore: An LLM-based metric (using GPT-4o) estimating the minimum line-by-line corrections needed to transform a candidate report into a ground-truth report.

Participing Models

The leaderboard currently features 16 models from 10 institutions, including specialized vision-language models (e.g., MedVersa, MAIRA-2, RadFM), instruction-tuned foundation models (e.g., CheXagent, GPT-4V), and models trained on specific datasets (e.g., VLCI, Cvt2distilgpt2).

Key Results

The evaluation of 16 models across the four datasets yields several key findings:

Top Performance: MedVersa emerges as the top-performing model, achieving the best $1/\text{RadCliQ-v1}$ scores across ReXGradient ( $1.01 \pm 0.01$ for Findings, $0.98 \pm 0.05$ for Findings+Impression), MIMIC-CXR ( $1.10 \pm 0.02$ for Findings, $0.92 \pm 0.02$ for Findings+Impression), and IU-Xray ( $1.46 \pm 0.03$ for Findings, $1.45 \pm 0.04$ for Findings+Impression). It consistently outperforms GPT-4V, a state-of-the-art generalist vision-language model, across multiple metrics and datasets.
Dataset Characteristics:
- IU-Xray is identified as the least challenging dataset, yielding consistently high performance scores across models, suggesting it may lack the complexity required for rigorous differentiation.
- CheXpert Plus exhibits the highest variance and lowest overall performance, likely due to its distinct data distribution and small validation set size.
- ReXGradient demonstrates remarkably low performance variance across models, underscoring its high data quality and utility as a reliable benchmark for assessing model robustness.
Training Data Impact: Models trained on multiple datasets (e.g., CheXpertPlus_CheX_MIMIC) tend to outperform those trained on single datasets, suggesting that multi-dataset training helps bridge distributional gaps. Conversely, models generally perform better when evaluated on the same distribution seen during training (e.g., VLCI_IU on IU-Xray vs. VLCI_MIMIC).
Task Complexity: Performance varies between generating "Findings" only versus "Findings + Impressions." For instance, MedVersa showed slight degradation when generating both sections on ReXGradient (dropping from $1.01 \pm 0.01$ to $0.98 \pm 0.05$ ), whereas CheXpertPlus_CheX_MIMIC showed improvement, potentially due to its use of separate specialized models for each subtask.

Significance and Claims

The paper claims that ReXrank provides a standardized evaluation framework that enables meaningful comparisons of model performance and offers crucial insights into model robustness across diverse clinical settings. By integrating a large-scale private dataset (ReXGradient) with public benchmarks and a suite of clinically relevant metrics, ReXrank moves beyond simple text similarity to assess the clinical accuracy and generalization capabilities of AI systems.

The authors position ReXrank not merely as a leaderboard for chest X-rays but as a foundational framework intended to set the stage for the comprehensive evaluation of automated reporting across the full spectrum of medical imaging. The inclusion of 95% confidence intervals in the results further emphasizes the statistical rigor of the evaluation, aiming to provide a reliable basis for future research and clinical adoption.