💬 NLP

PathReportEval: A Systematic Benchmark for Pathology Report Generation

This paper introduces PathReportEval, a standardized benchmark and evaluation framework for pathology report generation that addresses the limitations of existing lexical metrics by employing a clinically grounded Clinical Report Quality Score (CRQS) to accurately assess factual correctness, hallucinations, and clinical discordance across diverse models and datasets.

Original authors: Suryakant Singh, Sejuti Majumder, Beatrice Knudsen, Joel Saltz, Prateek Prasanna

Published 2026-07-22

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Suryakant Singh, Sejuti Majumder, Beatrice Knudsen, Joel Saltz, Prateek Prasanna

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers can look at a microscopic slide of human tissue and write a doctor's report about what's wrong. This is the cutting edge of "computational pathology," a field where artificial intelligence tries to translate gigapixel images of cells into the complex, life-saving language of medical diagnoses. For a computer to do this, it needs to be a master of two things: seeing the tiny details in the image (like spotting a suspicious cluster of cells) and writing a clear, accurate story about what it sees. But here's the tricky part: how do we know if the computer is actually doing a good job? If a computer writes a report that sounds fancy and uses the right medical words, does that mean it's correct? Or could it be confidently wrong, missing a cancer diagnosis or inventing a problem that doesn't exist? This is the big question scientists are wrestling with: how do we measure if a robot doctor is telling the truth, not just sounding like one?

Enter PathReportEval, a new study that acts like a strict, fair referee for these AI systems. The researchers realized that the current way of judging these computer-generated reports is broken. Right now, most scientists use standard "language games" (metrics like BLEU and ROUGE) that simply count how many words the computer's report shares with a human's report. It's like grading a student's essay only on how many of the teacher's favorite words they used, without checking if the student actually understood the math problem. The paper argues that this approach is dangerous because a computer could get a high score by copying the style of a report while completely missing the critical medical facts.

To fix this, the team built a massive, standardized testing ground. They took four different AI "students" and tested them on three different sets of real-world medical data, using three different "eyes" (visual encoders) to see the images. But the real star of the show is their new grading system, called the Clinical Report Quality Score (CRQS). Instead of just counting matching words, CRQS acts like a senior pathologist who reads the report and checks a specific checklist: Did the AI mention the diagnosis? Did it get the tumor grade right? Did it invent a fake cancer (a "hallucination")? Or did it contradict the facts?

The results were eye-opening. The study found that the old "word-counting" methods often gave high scores to reports that were actually medically dangerous or completely wrong. For example, one AI might get a high score for writing a report that sounded very similar to a real one, even if it accidentally changed a "high-grade" cancer to a "low-grade" one—a mistake that could cost a patient their life. In contrast, the new CRQS system caught these errors immediately, giving a low score to the wrong report and a high score to the one that got the facts right, even if the wording was different. The paper suggests that to make real progress in this field, we need to stop just looking at how "pretty" the text sounds and start measuring how clinically accurate it is. By providing this new, fair benchmark and a public toolkit for testing, the authors hope to help build AI systems that doctors can actually trust to help save lives.

Technical Summary: PathReportEval

Problem Statement

Pathology report generation from whole-slide images (WSIs) is a critical multimodal learning task, yet progress in this field is hindered by a lack of standardized evaluation. Existing studies suffer from heterogeneous experimental setups, utilizing different datasets, preprocessing pipelines, visual encoders, and decoding strategies, which makes direct comparison of model architectures unreliable. Furthermore, the field relies heavily on standard Natural Language Generation (NLG) metrics such as BLEU, ROUGE, and METEOR. These metrics measure lexical overlap but fail to assess clinical fidelity. A generated report may achieve high lexical similarity to a reference while omitting critical diagnoses, hallucinating findings, or stating incorrect tumor grades and stages—errors that are clinically consequential but linguistically subtle.

Methodology

The authors introduce PathReportEval, a systematic benchmark and evaluation framework designed to standardize the assessment of pathology report generation.

1. Standardized Benchmarking Framework

The framework evaluates four representative generation methods across three distinct datasets and three pathology foundation visual encoders under a unified protocol:

Datasets:
- TCGA (PathText): Large-scale, heterogeneous reports with higher annotation noise.
- HistAI: Multi-institutional, diverse tissue types with moderate structure.
- REG 2025: Shorter, semi-structured reports with high consistency and low noise.
Visual Encoders: Three foundation models representing different families: CONCHv1.5, UNI2-h, and H-Optimus-1.
Generation Methods:
- WSI-Caption (MI-Gen): Multiple-instance learning approach.
- HistGen: Hierarchical local-global encoding with cross-modal context.
- BiGen: Retrieval-augmented model using external knowledge.
- SCOUT: Context-aware transformer with concept conditioning.
Protocol: The authors re-implemented all methods in a shared PyTorch Lightning codebase to eliminate implementation variance. The pipeline standardizes preprocessing, tokenization, feature extraction (via TRIDENT), training, and decoding.

2. Clinical Report Quality Score (CRQS)

To address the limitations of lexical metrics, the paper proposes CRQS, a clinically grounded metric inspired by synoptic reporting practices (e.g., College of American Pathologists). Instead of comparing free text, CRQS maps both reference and generated reports into a structured schema of clinical attributes (e.g., diagnosis, grade, stage, margin status, lymph-node involvement).

CRQS decomposes report quality into four interpretable components:

Clinical Fact Coverage (CFC): The fraction of reference clinical fields correctly recovered.
Key Information Recall (KIR): The recovery rate of high-impact diagnostic attributes (diagnosis, grade, stage, etc.).
Hallucination Rate (HR): The proportion of generated facts not supported by the reference report.
Clinical Discordance Score (CDS): The rate of direct contradictions between generated and reference values for shared fields.

The final score is a weighted combination of these components ( $CRQS_{raw} = 0.30 \cdot CFC + 0.40 \cdot KIR + 0.20 \cdot (1-HR) + 0.40 \cdot (1-CDS)$ ), normalized to a range of [0, 1]. The weighting scheme prioritizes the recovery of critical information and penalizes clinical discordance heavily.

Key Results

The experiments reveal significant discrepancies between standard NLG metrics and clinical correctness:

Metric Misalignment: Conventional metrics (BLEU, ROUGE, METEOR) are weakly aligned with CRQS. Models with similar lexical scores often exhibit substantial differences in diagnostic accuracy.
- Example: In one case, a model downgraded a lesion from HSIL/CIN2 to LSIL/CIN1. METEOR assigned a moderate score (0.42), while CRQS correctly identified the critical error with a negative score (-0.07). Conversely, a model using different terminology for a benign condition received a near-zero METEOR score but a high CRQS score (0.91) for preserving clinical meaning.
Dataset Dependency: Performance varies significantly by dataset structure.
- TCGA: Exhibited lower CRQS and high hallucination rates (~0.5), suggesting lexical metrics overestimate quality on unstructured, noisy data.
- REG 2025: Showed stronger agreement between lexical metrics and CRQS, likely due to its semi-structured nature.
Encoder Sensitivity: No single visual encoder dominates across all datasets and metrics. For instance, UNI2-h performed best on REG 2025, while H-Optimus-1 showed strength on TCGA and HistAI for specific metrics. This highlights the need to report encoder effects explicitly rather than averaging them out.
Method Performance: Rankings shifted depending on the metric and dataset. For example, on REG 2025, SCOUT led in NLG metrics while BiGen achieved the highest CRQS.

Significance and Contributions

The paper argues that progress in pathology report generation must be measured by clinical correctness rather than lexical similarity. The authors' contributions are threefold:

Standardized Benchmark: A reproducible evaluation protocol across multiple datasets, encoders, and architectures, eliminating confounding variables from experimental setups.
Modular Framework: A public, plug-and-play framework (available at the provided GitHub repository) that allows for the rapid integration of new models, datasets, and encoders under a consistent pipeline.
CRQS Metric: A structured evaluation tool that moves beyond surface-level text overlap to assess factual correctness, hallucination, and clinical discordance.

The authors conclude that standard NLG metrics are insufficient proxies for clinical utility. By providing a framework that exposes failure modes such as omitted diagnoses and hallucinated findings, PathReportEval establishes a foundation for developing and evaluating models that are truly faithful to clinical requirements.