💻 computer science

Two-Stage Synthetic-to-Real Transfer Learning for Automated Mammography Report Generation Using Vision-Language Models

This paper proposes a two-stage synthetic-to-real transfer learning framework that leverages 400 clinically validated synthetic reports to pretrain a vision-language model, significantly improving automated mammography report generation performance and data efficiency while reducing hallucinations compared to existing state-of-the-art methods.

Original authors: Gani Esen, Marat Nurtas, Jandos Amankulov, Laura La Paglia

Published 2026-07-20

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Gani Esen, Marat Nurtas, Jandos Amankulov, Laura La Paglia

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to write a story, but there's a catch: you only have a few pages of the actual story to show it, and the rest of the library is empty. This is the daily struggle for scientists working on Automated Medical Report Generation. They want computers to look at medical pictures (like X-rays or mammograms) and write the doctor's notes that explain what they see. To do this, they use Vision-Language Models (VLMs), which are like digital brains that can "see" an image and "speak" in sentences. Usually, these brains learn by reading millions of picture-and-story pairs. But in the world of breast cancer screening, those pairs are incredibly rare. Most public databases have the pictures and some checkboxes (like "mass found" or "calcification"), but they are missing the free-flowing, human-written reports that the robot needs to learn the proper style and vocabulary. Without enough real examples, the robot either stays silent or starts making up wild, dangerous stories about things that aren't there.

This paper tackles that exact problem with a clever two-step strategy. The researchers, led by Gani Esen and Marat Nurtas, realized that instead of waiting for more real reports to appear, they could build a training camp using synthetic data. Think of it like a flight simulator: before a pilot flies a real plane, they practice in a simulator where the rules are perfect and the scenarios are generated by a computer. Here, the team used a powerful language model to turn simple checkboxes into fake, but medically plausible, reports. They then used these fake reports to give the robot a "head start" in learning the language of radiology. After the robot got comfortable with the basics in the simulator, they let it practice on the small pile of real doctor reports they actually had. The result? The robot learned much faster, made fewer dangerous mistakes, and wrote reports that sounded much more like a real doctor, even with very little real data to work with.

The Two-Stage Flight Simulator

The core idea of this research is a Two-Stage Synthetic-to-Real Transfer Learning framework. Let's break down how this works, step by step, using the analogy of training a new apprentice doctor.

The Problem: The Empty Library
In the real world, doctors use a specific checklist called BI-RADS to describe what they see in a mammogram. It's a structured system with categories for density, masses, and recommendations. Public datasets like VinDr-Mammo and CBIS-DDSM have thousands of these checklists, but they don't have the full, written reports. The only dataset with real, written reports is called DMID, but it's tiny—only about 225 cases. If you try to teach a giant AI brain (specifically a model called MedGemma-4B) to write reports using only those 225 examples, it gets confused. It either writes gibberish or hallucinates (makes up) findings that aren't there. In fact, without extra help, the AI would hallucinate findings about 73.1% of the time. That's a disaster for patient safety.

Stage 1: The Simulator (Synthetic Pretraining)
To fix this, the authors built a "simulator." They took the structured checklists from the big public datasets (VinDr-Mammo and CBIS-DDSM) and fed them into a smart language model (Llama-3.1-8B). This model acted like a ghostwriter, turning the checkboxes into 400 synthetic reports. These weren't real patient stories, but they followed the correct medical rules and vocabulary.

The Magic Trick: They used a technique called LoRA (Low-Rank Adaptation). Imagine the AI brain is a giant encyclopedia. Instead of rewriting the whole book, LoRA adds a small, flexible notebook on top of it. The team taught this notebook using the 400 fake reports. This taught the AI the structure of a mammography report: how to talk about breast density, how to list findings, and how to use the BI-RADS categories.
The Result: The AI learned the "grammar" of radiology without needing a single real report.

Stage 2: The Real World (Real Data Fine-Tuning)
Once the AI had its "notebook" filled with the basics from the simulator, they moved to the real world. They took that same notebook and fine-tuned it on the 407 real reports from the DMID dataset. Because the AI already knew the rules of the game from Stage 1, it only needed a little bit of real practice to learn the specific style and nuance of real doctors.

The Bonus: They also added a RAG (Retrieval-Augmented Generation) module. This is like giving the AI a medical dictionary (the ACR BI-RADS Atlas) that it can look up during the test. If the AI gets stuck, it can peek at the dictionary to make sure it's using the right clinical terms.

The Results: A Smarter, Safer Robot

When the researchers tested their new two-stage robot against the real DMID test set (52 cases), the results were impressive. They compared their method against nine other approaches, including robots that just guessed (zero-shot) and robots that tried to learn from scratch using only the real data.

The Scoreboard
The two-stage robot crushed the competition. Here is how it performed compared to the previous best method (called AMRG):

ROUGE-L: Improved by 17.9% (reaching 0.671).
METEOR: Improved by 11.4% (reaching 0.685).
CIDEr: Improved by 25.3% (reaching 0.729).
BERTScore: Reached 0.917.

These numbers might look like code, but they mean the robot's reports were much closer to what a human doctor would write. It wasn't just a little better; it was a significant leap forward.

The Safety Boost: Fewer Hallucinations
The most critical finding wasn't just about writing better sentences; it was about safety.

Zero-shot AI: When asked to write a report without any training, the AI made up findings 73.1% of the time. It would confidently describe tumors that didn't exist.
Two-stage AI: After the two-stage training, the hallucination rate dropped to 21.2%.
The "Good" Mistake: Even when the two-stage AI did make a mistake, it was often a "safer" one. For example, in one case where the breast was actually normal, the AI trained only on real data predicted a high-risk cancer (BI-RADS 4c), which would lead to an unnecessary biopsy. The two-stage AI predicted a "probably benign" result (BI-RADS 3), which just means "let's check it again later." While still not perfect, the two-stage model made a less harmful error.

Data Efficiency: Doing More with Less
One of the most exciting discoveries was how much real data the AI actually needed. The researchers tested the system with fewer and fewer real examples.

They found that the two-stage approach reduced the need for real data by about 2 times.
With just 100 real examples, the two-stage model was already better than the previous state-of-the-art model that used all 407 examples.
This suggests that in the future, hospitals with very few labeled reports could still train powerful AI tools, provided they use this synthetic pretraining trick.

What This Means (and What It Doesn't)

The authors are careful to note that this is a significant step forward, but not a finished product. The AI is still not perfect; its accuracy in predicting the final BI-RADS category was around 31%, which shows that reading mammograms is still incredibly hard for computers. The study also points out that the test set was small (only 52 cases), so larger studies are needed to be sure.

However, the paper explicitly rules out a few things:

Just using big models isn't enough: Simply taking a giant AI like Qwen or LLaVA and asking it to write reports without this specific training didn't work. They performed terribly.
Old methods aren't the answer: Traditional "pipeline" methods (where you use one AI to find the object and another to write the text) were competitive but couldn't match the coherence of the two-stage end-to-end approach.
Preprocessing isn't the key: The team tried fancy image cleaning techniques (like adjusting contrast or cropping), but found that the AI actually worked better with the raw images, suggesting the AI's vision system is already quite robust.

The Takeaway

This paper suggests that when you don't have enough real-world data to train a medical AI, you can build a bridge using synthetic data. By teaching the AI the rules of the game in a simulator first, it becomes much more efficient and safer when it finally enters the real world. The authors released all their code, synthetic datasets, and model weights, inviting other scientists to try this "two-stage" approach on other medical problems where data is scarce. It's a playful, practical solution to a very serious problem: how to teach a robot to be a good doctor's assistant when there aren't enough human teachers to go around.

Technical Summary: Two-Stage Synthetic-to-Real Transfer Learning for Automated Mammography Report Generation

Problem Statement

The development of Vision-Language Models (VLMs) for automated mammography report generation is severely constrained by the scarcity of publicly available paired datasets containing mammographic images and corresponding free-text radiology reports. While structured annotations (e.g., BI-RADS categories, lesion locations) exist in datasets like VinDr-Mammo and CBIS-DDSM, they lack the narrative text required for supervised training. The only major public resource with paired data, the Digital Mammography Dataset for Breast Cancer Diagnosis Research (DMID), is limited to 225 cases (510 images). This data scarcity hinders the training of models capable of generating clinically accurate, structured reports that adhere to the American College of Radiology's BI-RADS lexicon, a necessity for standardizing breast cancer screening and reducing radiologist cognitive burden.

Methodology

The authors propose a two-stage synthetic-to-real transfer learning framework designed to overcome data limitations by decoupling domain learning into complementary phases. The framework utilizes the MedGemma-4B model, comprising a SigLIP vision encoder and a Gemma-3 language backbone, adapted via Low-Rank Adaptation (LoRA).

1. Label-to-Report Synthesis Pipeline (Stage 1)

To generate pretraining data, the authors developed a pipeline that converts structured annotations from VinDr-Mammo and CBIS-DDSM into free-text reports.

Input: Structured annotations including laterality, view position, ACR density, finding categories, BI-RADS assessment, and bounding boxes.
Generation: A structured prompt is fed into Llama-3.1-8B (via local inference) to generate synthetic reports.
Validation: Generated reports undergo a strict quality validation function $V(\hat{r}_i)$ ensuring the presence of BI-RADS mentions, recommendations, a minimum length of 25 words, and semantic consistency with the input annotations.
Output: A curated dataset of 400 synthetic image-report pairs.

2. Two-Stage Training Paradigm

Stage 1 (Synthetic Pretraining): The MedGemma-4B model is fine-tuned on the 400 synthetic pairs using multimodal LoRA. This phase teaches the model the specific vocabulary, structure, and clinical conventions of mammography reporting.
Stage 2 (Real-Data Fine-Tuning): The LoRA weights pretrained in Stage 1 are transferred and further fine-tuned on the 407 training examples from the real DMID dataset. This stage refines the model's output style to match authentic radiologist language and corrects any synthetic artifacts.

3. BI-RADS-Grounded Retrieval-Augmented Generation (RAG)

To provide domain-specific clinical context during inference, a RAG module is integrated.

Knowledge Base: Consists of 39 text passages from the ACR BI-RADS Atlas.
Mechanism: At inference, the system encodes the input and retrieves the most relevant passage using Sentence-BERT embeddings and a FAISS index, grounding the generation in established clinical guidelines.

4. Implementation Details

Architecture: MedGemma-4B with LoRA applied to seven target modules per transformer block ( $W_q, W_k, W_v, W_o, W_{gate}, W_{up}, W_{down}$ ).
Quantization: The base model is quantized to 4-bit (NF4) using QLoRA to enable training on a single NVIDIA RTX 5090 GPU.
Hyperparameters: The optimal configuration identified via ablation is rank $r=64$ and scaling factor $\alpha=64$ for Stage 2, with Stage 1 fixed at $r=16, \alpha=32$ .

Key Contributions

Two-Stage Transfer Learning Framework: A novel approach that pretrains on LLM-generated synthetic reports before fine-tuning on limited real data, demonstrating statistically significant improvements ( $p=0.045$ ) over training on real data alone.
Scalable Synthesis Pipeline: An automated, BI-RADS-compliant pipeline that converts structured annotations from public datasets (VinDr-Mammo, CBIS-DDSM) into validated synthetic training data, effectively expanding the available training corpus.
Comprehensive Benchmarking: Evaluation against nine baselines, including zero-shot VLMs (Qwen2.5-VL, LLaVA-1.6), pipeline approaches (CLIP+GPT2), and the previous state-of-the-art (AMRG).
Data Efficiency Analysis: Demonstration that synthetic pretraining reduces real data requirements by approximately 2×, with the two-stage model trained on only 100 real examples surpassing the AMRG benchmark.
Clinical Quality Improvements: Significant reduction in hallucination rates and improved adherence to report structure and length.

Results

Evaluated on the DMID test set ( $n=52$ ), the two-stage approach achieved the following metrics, outperforming all baselines:

BLEU-4: 0.312
ROUGE-L: 0.671 (vs. 0.569 for AMRG, a +17.9% improvement)
METEOR: 0.685 (vs. 0.615 for AMRG, a +11.4% improvement)
CIDEr: 0.729 (vs. 0.582 for AMRG, a +25.3% improvement)
BERTScore: 0.917

Clinical Metrics:

Hallucination Rate: Reduced from 73.1% (zero-shot MedGemma-4B) to 21.2% (two-stage model).
Report Length: The model generated reports averaging 42.2 words, closely matching the 43.1-word average of real radiologist reports, compared to 129.2 words for the zero-shot baseline.
Density Prediction: The two-stage model showed improved accuracy in predicting ACR breast density (4/5 correct in qualitative analysis) compared to the DMID-only baseline (1/5), suggesting successful knowledge transfer from synthetic pretraining.

Significance and Claims

The paper claims that this work addresses the fundamental bottleneck of data scarcity in medical VLM development for mammography. By leveraging synthetic data as a pretraining curriculum, the authors demonstrate that:

Synthetic data is effective: LLM-generated reports serve as high-quality pretraining data that teaches domain-specific structure and vocabulary, enabling more efficient learning on small real datasets.
Data efficiency is critical: The framework allows high-performance report generation with significantly fewer real-world paired examples, making it viable for clinical settings where data is scarce.
Clinical safety: The approach significantly reduces hallucinations and produces reports that are structurally and linguistically closer to human radiologist outputs, even when the underlying clinical content is imperfect.
Generalizability: The synthetic-to-real transfer learning paradigm is proposed as a general strategy for low-resource medical AI development beyond mammography.

The authors emphasize that while the DMID test set is small, the systematic ablation of LoRA hyperparameters and the consistent outperformance of diverse architectural paradigms (zero-shot, pipeline, and direct fine-tuning) validate the robustness of the proposed method. The work concludes that this framework provides a practical, reproducible foundation for automated mammography reporting, with all code, synthetic datasets, and model weights released for reproducibility.