💻 computer science

CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographs

The paper introduces CheXGenBench, a unified benchmark framework that evaluates the fidelity, privacy, and clinical utility of synthetic chest radiographs across 11 text-to-image models using over 20 standardized metrics, while also releasing a 75K-image synthetic dataset (SynthCheX-75K) to advance medical AI research.

Original authors: Raman Dutt, Pedro Sanchez, Yongchen Yao, Steven McDonagh, Sotirios A. Tsaftaris, Timothy Hospedales

Published 2026-03-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Raman Dutt, Pedro Sanchez, Yongchen Yao, Steven McDonagh, Sotirios A. Tsaftaris, Timothy Hospedales

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to teach a robot to cook perfect meals. You have a massive library of real recipes and photos of dishes (the real medical data), but you can't share the actual photos because they contain private information about real people. So, you ask the robot to imagine new photos of dishes based on descriptions like "a plate with a broken bone" or "a healthy lung."

This paper is about building a super-tough taste test (a benchmark) to see if these robot-generated photos are good enough to actually help doctors, or if they are just fancy forgeries that could get people in trouble.

Here is the breakdown of CheXGenBench using simple analogies:

1. The Problem: The "Fake News" of Medical Images

In the world of AI, robots have gotten really good at making fake images that look real. But in medicine, "good enough" isn't enough.

The Risk: If a robot makes a fake X-ray that looks too much like a real patient's X-ray, it could accidentally reveal who that patient is (a privacy leak).
The Usefulness: If the fake X-ray looks like a healthy lung but the robot is supposed to show a broken bone, it's useless for training doctors.
The Mess: Until now, scientists were testing these robots with different rulers, different measuring cups, and outdated recipes. Some said, "Great job!" while others said, "Terrible!" It was impossible to know who was actually the best chef.

2. The Solution: CheXGenBench (The Ultimate Taste Test)

The authors built CheXGenBench, a unified "scorecard" that tests these AI chefs on three specific things, like a judge at a cooking competition:

🏆 Category 1: Fidelity (Does it look real?)

The Analogy: Imagine looking at a fake diamond. Is it shiny enough to fool the eye?
The Test: The benchmark checks if the AI's fake X-rays look like real X-rays. But they didn't just use an old ruler (an old AI model); they used a super-advanced medical eye (a new AI called RadDino) to spot tiny details.
The Surprise: They found that many popular AI models were actually quite bad at making X-rays of rare diseases. They were great at making pictures of "healthy lungs" (because that's what they saw most often in training) but terrible at making pictures of rare, complex conditions.

🔒 Category 2: Privacy (Is it a spy?)

The Analogy: Imagine the robot is a spy. If you ask it to draw "a picture of a patient with a broken leg," does it accidentally draw your specific broken leg from your medical file?
The Test: They tried to match the fake images back to the real patients the robot was trained on.
The Scary Finding: Even the "best" robots were leaking secrets. About 10% to 25% of the fake images were so similar to real patients that a computer could identify the specific person. It's like the robot memorized the recipe book instead of learning to cook.

🛠️ Category 3: Utility (Is it actually useful?)

The Analogy: If you give a student a textbook full of fake photos, will they pass the medical exam?
The Test: They took the fake X-rays and used them to train a different AI to diagnose diseases.
The Result:
- Good News: For common diseases, the fake images worked almost as well as real ones!
- Bad News: For rare diseases, the fake images didn't help much. The AI students failed the exam on the hard questions.
- Report Cards: When asked to write a medical report based on the fake X-ray, the AI struggled to get the facts right, often hallucinating (making up) symptoms.

3. The Winners and Losers

The paper tested 11 different AI models (the "chefs").

The Star Chef: A model called Sana (specifically a 0.6 billion parameter version) won the competition. It made the most realistic images, leaked the least amount of private info, and was the most helpful for training other AIs.
The Old Guard: Surprisingly, some older, famous models (like early versions of Stable Diffusion) performed poorly, even though they are popular in other fields.
The Big Models: Bigger isn't always better. Some massive models (12 billion parameters) performed worse than the smaller, more efficient ones because they weren't tuned correctly for medical data.

4. The Gift: SynthCheX-75K

Because the "Sana" model was so good, the authors used it to create a massive new library of 75,000 fake X-rays.

They cleaned this library to remove the "bad apples" (low-quality or unsafe images).
They released this library to the public for free.
Why? So other researchers can train their own medical AIs without needing to access real, private patient data. It's like giving the world a safe, open-source cookbook.

The Big Takeaway

This paper is a wake-up call. It says: "Stop just making pretty pictures. We need to test if these medical fakes are safe, accurate, and actually useful."

They provided the ruler (CheXGenBench) to measure quality, the recipe (training protocols) to make better models, and the ingredients (the 75K dataset) to help everyone cook up better medical AI in the future.

In short: They built a rigorous test to ensure that when AI creates fake medical images, it's a helpful tool for doctors, not a privacy nightmare or a useless toy.

1. Problem Statement

The generation of synthetic medical images via Text-to-Image (T2I) models holds promise for addressing data scarcity and privacy concerns in healthcare. However, the field suffers from a "benchmarking crisis" characterized by:

Methodological Inconsistencies: Lack of standardized training protocols and data splits across studies.
Outdated Evaluation Metrics: Heavy reliance on Fréchet Inception Distance (FID) using encoders trained on natural images (InceptionV3) or outdated medical backbones (DenseNet-121), which fail to capture medical nuances.
Disconnected Criteria: Studies typically evaluate either image fidelity or privacy risks, rarely addressing both simultaneously or assessing downstream clinical utility.
Limited Scope: Most evaluations focus on global averages, ignoring the long-tailed distribution of medical pathologies (where rare diseases are critical but underrepresented).
Architectural Lag: Many studies use outdated T2I architectures, failing to assess how modern foundation models perform in the medical domain.

2. Methodology: CheXGenBench

The authors introduce CheXGenBench, a unified framework designed to evaluate T2I models across three critical dimensions: Fidelity, Privacy, and Utility.

A. Experimental Setup

Dataset: MIMIC-CXR (237,388 training samples, 5,034 test samples).
Prompt Engineering: The authors adopted LLaVA-Rad annotations (enhanced, descriptive captions) instead of original MIMIC impressions. They empirically proved that LLaVA-Rad improves fidelity and reduces re-identification risks.
Models Evaluated: 11 leading T2I architectures, stratified by size:
- Small (<1B params): SD V1-4/5, SD V2/2-1, RadEdit, Pixart-Sigma, Sana.
- Large (>1B params): SD V3.5, Lumina 2.0, Flux.1-Dev, LLM-CXR.
Training Protocol: Standardized training (20 epochs) with Full Fine-Tuning (FFT) for small models and LoRA (Rank 32) for large models to ensure fair comparison.

B. Evaluation Dimensions & Metrics

The benchmark employs 20+ quantitative metrics:

Generative Fidelity & Mode Coverage:
- FID/KID: Calculated using RadDino (a state-of-the-art medical vision encoder) instead of InceptionV3 or DenseNet-121 to ensure domain alignment.
- Image-Text Alignment: Measured using BioViL-T (biomedical vision-language model).
- Mode Coverage: Uses Precision, Recall, Density, and Coverage (PRDC) metrics to evaluate if models capture the diversity of the data distribution, specifically addressing long-tailed pathologies.
- Conditional Analysis: Metrics are computed independently for 14 distinct pathologies (e.g., Pneumonia, Fracture, No Finding) to detect performance gaps in rare conditions.
Privacy & Re-Identification Risk:
- Assesses if models memorize training data.
- Metrics:
  - Pixel Distance: $\ell_2$ distance between synthetic and real images.
  - Latent Distance: Euclidean distance in RadDino embedding space.
  - Re-Identification Score: Probability of matching a synthetic image to a real patient using a Siamese network (ResNet-50 backbone).
Downstream Clinical Utility:
- Image Classification: A ResNet-50 classifier is trained only on synthetic data and tested on real data. Metrics: Accuracy, F1-Score, AUROC.
- Radiology Report Generation (RRG): Fine-tuning a multimodal model (LLaVA-Rad) on synthetic data to generate reports. Metrics: BLEU, ROUGE-L, F1-RadGraph.

3. Key Contributions

Unified Benchmark: First framework to simultaneously evaluate fidelity, privacy, and utility for medical T2I generation.
Standardized Protocol: Established consistent training splits, prompt engineering (LLaVA-Rad), and evaluation metrics (RadDino-based) to enable reproducible comparisons.
SynthCheX-75K Dataset: Released a high-quality dataset of 75,000 synthetic radiographs generated by the top-performing model (Sana), filtered for quality using a medical VLM (HealthGPT).
Comprehensive Analysis: Provided the first pathology-conditional analysis, revealing how model performance correlates with pathology prevalence.

4. Key Results & Findings

A. Generative Fidelity

Top Performer: Sana (0.6B) achieved the best overall performance, with the lowest FID (54.22) and KID scores, and highest Recall/Coverage. It successfully generated diverse pathological modes.
Runner-up: Pixart-Sigma showed strong image-text alignment and second-best fidelity.
Failure of Large Models: Surprisingly, larger models (SD V3.5, Flux.1-Dev, Lumina 2.0) often underperformed smaller models when using LoRA, suggesting that parameter-efficient fine-tuning may be insufficient for the complex medical domain without full adaptation.
Long-Tail Problem: All models struggled with rare pathologies (e.g., "Pleural Other"). Performance strongly correlated with pathology frequency in the training set (correlation coefficient 0.947). Models excelled at "No Finding" (healthy) but failed on rare diseases.

B. Privacy Risks

High Risk Across the Board: Despite varying fidelity, all models posed significant re-identification risks.
Memorization: A substantial proportion of samples (10%–25%) could be re-identified with high confidence (Re-ID score > 0.85).
LoRA Benefit: Models trained with LoRA showed slightly lower memorization risks compared to full fine-tuning, likely due to reduced capacity to memorize specific training examples.

C. Downstream Utility

Classification: Sana was the only model whose synthetic data allowed a classifier to match or exceed the performance of a classifier trained on real data for 10 out of 14 pathologies (including under-represented classes like Fracture).
Report Generation: Synthetic data generally degraded report generation performance compared to real data. No single model dominated all RRG metrics, indicating that current T2I models lack the necessary image-text alignment for complex clinical reasoning tasks.

5. Significance and Impact

New State-of-the-Art: Establishes Sana as the current SoTA for synthetic chest radiograph generation, demonstrating that efficient, smaller models can outperform massive architectures in medical domains with proper tuning.
Clinical Relevance: Proves that high-fidelity synthetic data can serve as a viable standalone replacement for real data in training diagnostic classifiers, potentially solving data scarcity issues.
Privacy Warning: Highlights that high fidelity does not equate to privacy; even the best models leak patient information, necessitating stronger anonymization techniques (e.g., differential privacy) before clinical deployment.
Community Resource: The release of CheXGenBench and SynthCheX-75K provides the medical AI community with a rigorous, reproducible standard for evaluating future generative models, moving the field away from fragmented, non-comparable studies.

In conclusion, CheXGenBench reveals that while generative AI has made strides in medical image synthesis, significant challenges remain regarding long-tail pathology generation and patient privacy. The framework serves as a critical tool for guiding future research toward models that are not only visually realistic but also clinically useful and safe.