💬 NLP

Improving Automatic Summarization of Radiology Reports through Mid-Training of Large Language Models

This paper demonstrates that a "pre-training, mid-training, fine-tuning" strategy using large-scale clinical text significantly improves the automatic summarization of radiology reports, achieving superior performance in both text quality and factuality compared to traditional direct fine-tuning approaches.

Original authors: Mengxian Lyu, Cheng Peng, Ziyi Chen, Mengyuan Zhang, Jieting Li Lu, Yonghui Wu

Published 2026-03-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mengxian Lyu, Cheng Peng, Ziyi Chen, Mengyuan Zhang, Jieting Li Lu, Yonghui Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching a Doctor's Assistant

Imagine you have a brilliant, well-read assistant (a Large Language Model or LLM) who knows a lot about medicine. You want to teach this assistant a very specific job: reading long, messy X-ray reports and writing a short, clear summary for the doctor.

The problem is that X-ray reports are written in a weird, shorthand language full of jargon, abbreviations, and specific patterns that are very different from regular medical notes or general English.

This paper argues that the old way of training these assistants isn't working perfectly. Instead of just "teaching them the job" immediately, we need to give them a specialized internship first. The authors call this "Mid-Training."

The Three Steps of Training (The "School" Analogy)

The researchers tested three different ways to train their AI models. Think of it like a student's education path:

1. The Old Way: General School → Job Training (Pre-training + Fine-tuning)

The Process: The AI learns general English and broad medical facts (like reading every textbook in a library). Then, you immediately throw it into the job, showing it X-ray reports and saying, "Summarize this."
The Result: It's okay, but the AI is still confused by the specific "slang" of radiology. It's like hiring a generalist lawyer and asking them to immediately interpret a complex engineering blueprint without any engineering background.

2. The New Way: General School → Radiology Internship → Job Training (Pre-training + Mid-training + Fine-tuning)

The Process:
1. Pre-training: The AI learns general medicine (the library).
2. Mid-training (The Secret Sauce): Before doing the actual job, the AI spends time reading only thousands of X-ray reports. It doesn't have to summarize them yet; it just reads them to get used to the rhythm, the abbreviations, and the weird sentence structures. It's like a medical student doing a 6-month rotation specifically in the radiology department.
3. Fine-tuning: Finally, the AI is taught the specific task of summarizing.
The Result: Because the AI already "speaks the language" of radiology, it learns the summarization task much faster and does a much better job.

Why This Matters: The "Cold Start" Problem

The paper highlights a major issue called the "Cold Start" problem.

The Scenario: Imagine you only have 5 examples of X-ray reports to teach your AI.
Without Mid-training: The AI is like a tourist dropped in a foreign country with no phrasebook. It looks at the 5 examples and says, "I have no idea what's going on." It performs terribly because it doesn't understand the context.
With Mid-training: The AI is like a tourist who already studied the language for a month. Even with only 5 examples, it looks at them and says, "Ah, I recognize this pattern! I can figure this out."

The Finding: The "Mid-trained" models were able to learn effectively with very few examples (few-shot learning), while the others struggled until they were given hundreds of examples.

The "Fact Check" Bonus

It's not just about sounding good; it's about being factually correct.

In radiology, saying "No pneumonia" is very different from saying "Pneumonia present."
The researchers found that the Mid-trained models were much better at getting the medical facts right (measured by something called RadGraph-F1). They didn't just sound like a doctor; they actually understood the medical logic.

The "Scale" Surprise

Usually, in AI, bigger is better. If you have a massive model (3 billion parameters), it usually beats a small one.

The Twist: The researchers found that a small model that went through the "Mid-training" internship performed almost as well as a massive model that didn't get the internship.
Why this is cool: This means hospitals might not need super-expensive, massive computers to get great results. They can use smaller, cheaper models if they just give them that specialized "Mid-training" first.

Summary in One Sentence

To make AI better at summarizing X-ray reports, don't just throw it into the deep end; give it a "Mid-training" internship where it reads thousands of reports first, so it learns the specific language and logic before it tries to do the actual job.

1. Problem Statement

Automatic summarization of radiology reports (condensing detailed "Findings" into concise "Impressions") is critical for reducing physician burnout. While Large Language Models (LLMs) have been applied to this task, current approaches typically rely on a standard "Pre-training $\rightarrow$ Fine-tuning" pipeline.

The Gap: General medical pre-training (e.g., on discharge summaries or PubMed) often fails to capture the unique linguistic characteristics of radiology reports, such as omitted function words, dense anatomical jargon, and specific negation patterns.
The Challenge: Directly fine-tuning general or broad clinical models on radiology data often leads to suboptimal performance, particularly in few-shot learning scenarios (where labeled data is scarce), resulting in a "cold start" problem where models struggle to learn from small datasets.

2. Methodology

The authors propose a three-stage hierarchical adaptation pipeline: Pre-training $\rightarrow$ Mid-training $\rightarrow$ Fine-tuning.

A. Model Architecture

The study utilizes the T5 (Text-to-Text Transfer Transformer) architecture as the backbone. Three model scales were evaluated: 0.2B (Base), 0.7B (Large), and 3B (XL) parameters.

B. Training Stages

Clinical Pre-training (Foundation):
- Model: GatorTronT5.
- Data: >90 billion tokens comprising de-identified clinical notes from UF Health, PubMed, Wikipedia, and MIMIC-III.
- Objective: Unsupervised span-corruption denoising to establish broad clinical knowledge.
Subdomain Mid-training (The Novel Contribution):
- Model: GatorTronT5-Radio.
- Data: MIMIC-CXR (13M tokens from 227k radiographic studies).
- Process: The pre-trained GatorTronT5 is further trained on radiology reports using unsupervised span corruption. The "Findings" and "Impression" sections are concatenated to allow the model to internalize radiology-specific syntax, terminology, and structural priors before any supervised task is introduced.
Supervised Fine-tuning:
- Data: OpenI dataset (Indiana University Chest X-ray Collection), serving as an out-of-distribution (OOD) test set to evaluate generalizability.
- Task: Conditional sequence-to-sequence generation (Input: Findings $\rightarrow$ Output: Impression).

C. Experimental Design

Baselines: Compared against a general-purpose T5 and the clinically pre-trained GatorTronT5 (without mid-training).
Few-Shot Learning: Evaluated performance on subsets of the OpenI training data ranging from $k=5$ to $k=1000$ samples to test data efficiency and the "cold start" phenomenon.
Hardware: Training conducted on NVIDIA A100/B200 clusters using PyTorch and Hugging Face Transformers.

3. Key Contributions

Proposal of Mid-Training for Subdomain Adaptation: The study formalizes "mid-training" as a distinct intermediate stage to bridge the gap between broad clinical knowledge and specialized radiology syntax, arguing that direct fine-tuning is insufficient for high-specialty domains.
Alleviation of the "Cold Start" Problem: Demonstrated that mid-training significantly improves few-shot learning capabilities, allowing models to learn effectively from very small datasets (e.g., 5–20 samples) where non-mid-trained models fail.
Cross-Institutional Generalization: Validated the approach using a pipeline where the model is pre-trained at UF Health, mid-trained on MIMIC-CXR (Boston), and fine-tuned/evaluated on OpenI (Indiana), proving robustness across different institutions.

4. Results

The experiments were evaluated on the OpenI dataset using lexical overlap (ROUGE-L, METEOR), semantic similarity (BERTScore), and clinical factual consistency (RadGraph-F1).

Performance Gains: The mid-trained model (GatorTronT5-Radio) consistently outperformed both the general T5 and the clinical-only GatorTronT5 across all model scales.
- Example (0.2B Model): ROUGE-L improved from 0.4874 (General) $\rightarrow$ 0.5018 (Clinical) $\rightarrow$ 0.5281 (Mid-trained).
- Example (3B Model): The best performing model, GatorTronT5-XL-Radio, achieved a ROUGE-L of 0.6362 and RadGraph-F1 of 0.5655.
Factual Consistency: Mid-training significantly improved RadGraph-F1 scores, indicating better preservation of clinical facts and entities (e.g., correctly identifying negations and anatomical relations).
Few-Shot Learning:
- Cold Start Mitigation: With only 5 training samples, the non-mid-trained model achieved a ROUGE-L of 0.0992, while the mid-trained model achieved 0.4716.
- Threshold Effect: Non-mid-trained models exhibited a "threshold" behavior, requiring ~200 samples to show significant improvement. Mid-trained models showed immediate gains even with minimal data.
Efficiency: A smaller mid-trained model (0.7B) could rival the performance of a much larger non-mid-trained model (3B), suggesting high efficiency for low-resource settings.

5. Significance

Paradigm Shift: The study advocates for shifting the standard LLM adaptation pipeline from "Pre-train $\rightarrow$ Fine-tune" to "Pre-train $\rightarrow$ Mid-train $\rightarrow$ Fine-tune" for specialized medical domains.
Clinical Utility: By improving factual consistency (RadGraph-F1) and reducing the data required for effective deployment, this method makes AI summarization tools more viable for real-world radiology workflows, potentially reducing radiologist burnout.
Scalability: The findings suggest that subdomain-specific mid-training is a cost-effective strategy to enhance model performance without needing to retrain massive foundation models from scratch.