Technical Summary: Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports

Problem Statement

Reliable evaluation of generated radiology reports is critical because omitted critical findings or mischaracterized observations can directly impact patient care. Existing evaluation metrics, whether traditional lexical scores (e.g., BLEU, ROUGE) or embedding-based measures (e.g., BERTScore), reduce report quality to a single scalar value. This approach fails to distinguish between clinically significant errors (which alter diagnosis or treatment) and clinically insignificant variations (such as stylistic rephrasing or synonymous anatomical descriptions).

Recent studies indicate that even Large Language Models (LLMs) used as evaluators struggle to draw a reliable boundary between these two categories. While LLMs possess rich medical knowledge, they often exhibit a "discrimination bias": they effectively detect true clinical errors but simultaneously over-penalize harmless rephrasings, resulting in high sensitivity (Discrimination) but low tolerance for benign variations (Robustness).

Methodology

1. Evaluation Framework: ReEvalMed

The authors utilize the ReEvalMed benchmark, a meta-evaluation dataset comprising 400 report pairs derived from MIMIC-CXR. The benchmark is structured around two complementary dimensions:

• Discrimination (D): The ability to detect errors that materially affect clinical decisions (e.g., changing a lesion from benign to malignant).
• Robustness (R): The ability to remain unpenalized by clinically harmless variations (e.g., synonymous wording).

The evaluation treats the task as a classification problem rather than a scalar scoring task. The goal is to correctly label discrepancies as either "clinically significant" or "clinically insignificant."

2. LLM-as-Evaluator Paradigms

The study investigates two inference strategies for LLM-based evaluators:

• One-Pass: The model receives the reference (REF) and target (TGT) reports in a single prompt and directly outputs a structured JSON object containing error spans, aspects, and severity levels (Critical, Significant, Insignificant).
• Two-Pass: Inference is decoupled into two steps:
  1. Span Detection: Identifying all discrepancies between REF and TGT.
  2. Severity Judgment: Assigning a severity label to each detected span based on specific criteria.

3. Data Synthesis

To address the lack of supervised data for the clinical significance boundary, the authors synthesized a 4,000-sample dataset of radiology report pairs.

• Process: Using stratified sampling from MIMIC-CXR, they paired reference reports with controlled error injections.
• Annotation: Errors were categorized across 12 aspects (e.g., Location, Severity, Negation) and three types (Omission, Fabrication, Inaccuracy).
• Balance: The dataset contains an equal split of Discrimination (significant errors) and Robustness (insignificant variations) samples.
• Quality Control: A qualified clinician reviewed 400 randomly sampled pairs, achieving a 93% pass rate.

4. Metric Training

The authors trained lightweight, interpretable metrics on the synthesized data using two base models: Qwen3-8B and MedGemma-4B.

• Supervised Fine-Tuning (SFT): Taught the models to produce structured outputs (JSON) and perform basic error detection.
• Reinforcement Learning (RL): Specifically, Direct Preference Optimization (DPO) was applied to explicitly optimize the decision boundary between significant and insignificant errors. Preference pairs were constructed by treating the ground-truth severity label as the "chosen" response and a flipped label as the "rejected" response.

Key Results

Baseline Performance

• Score-based metrics: Traditional metrics (BLEU, BERTScore, RadGraph) performed poorly, with maximin accuracy scores generally below 55% on both D and R dimensions, confirming their inability to separate significant errors from noise.
• LLM-as-Evaluators (Zero-shot): Most LLMs (including proprietary models like GPT-5.1 and open-source models like Qwen3-Max) achieved high Discrimination (often >90%) but very low Robustness (often <60%). This confirmed a widespread discrimination bias.
• Two-Pass vs. One-Pass: Switching to a two-pass inference strategy did not consistently improve overall performance. Instead, it redistributed errors; for some models, it improved Robustness at the cost of Discrimination, and for others, it exacerbated the imbalance.

Performance of Trained Metrics

The trained lightweight metrics demonstrated significant improvements:

• Qwen3-8B (Two-Pass): Achieved 78.5% Discrimination and 70.5% Robustness (Gap: 8.0).
• MedGemma-4B (One-Pass): Achieved 75.5% Discrimination and 80.0% Robustness (Gap: 4.5).
• Comparison: These results surpassed 32B-scale medical LLMs (e.g., LingShu-32B, Hulu-Med-32B) and remained competitive with proprietary models like Claude Sonnet 4.5, despite using models with significantly fewer parameters.

Analysis of Training Stages

• SFT successfully taught the output format and improved basic detection but did not fully resolve the D-R imbalance.
• RL (DPO) was crucial for narrowing the gap between Discrimination and Robustness, effectively calibrating the model's sensitivity to the clinical significance boundary.

Significance and Claims

The paper claims three primary contributions:

1. Systematic Analysis of Bias: It reveals a consistent "discrimination bias" across current LLM evaluators, where models prioritize error detection over tolerating harmless variations, undermining their reliability as clinical indicators.
2. Clinically Grounded Data Synthesis: It introduces a balanced, 4k-sample dataset annotated with fine-grained error specifications, verified by clinicians, to explicitly supervise the boundary between significant and insignificant variations.
3. Lightweight Interpretable Metrics: It demonstrates that training small, open-source models (4B–8B parameters) on this synthesized data via SFT and RL can produce evaluators that outperform larger 32B medical models and rival proprietary systems.

Practical Implications:
The authors suggest that one-pass trained metrics are the practical choice for cost-sensitive deployments, offering a favorable balance of performance and efficiency. The two-pass setting is reserved for scenarios where the trade-off between discrimination and robustness is critical, though it does not eliminate errors but rather redistributes them.

Limitations:
The authors modestly note that full-scale clinician review of the synthesized dataset was not feasible due to cost, relying instead on a 400-sample random verification. Additionally, the training algorithm (DPO) was chosen for simplicity and reproducibility rather than as a novel optimization method, and the models' generalization to non-US clinical settings or other imaging modalities remains an open question.