💻 computer science

Cross-Modal Clinical Knowledge Integration for Mammography Report Generation

The paper proposes MammoRG, a two-stage framework that integrates structured clinical knowledge and BI-RADS guidelines to generate mammography reports with superior diagnostic consistency, accompanied by a dedicated parsing tool (MammoRGTool) for evaluating clinical efficacy.

Original authors: Jiayi Zhu, Fuxiang Huang, Yu Xie, Xi Wang, Zhixuan Chen, Yuan Guo, Qingcong Kong, Zhenhui Li, Qiong Luo, Hao Chen

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiayi Zhu, Fuxiang Huang, Yu Xie, Xi Wang, Zhixuan Chen, Yuan Guo, Qingcong Kong, Zhenhui Li, Qiong Luo, Hao Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The Radiologist's Overload

Imagine a radiologist is like a highly skilled detective who looks at X-ray pictures of breasts (mammograms) to find clues about cancer. Every day, they have to look at thousands of these pictures and write a detailed report for each one. It's a massive job, and it's easy to get tired or make small mistakes.

The paper points out that while computers have gotten very good at looking at pictures and writing simple descriptions, they are still terrible at acting like a real radiologist. If you ask a standard AI (like a generic chatbot) to look at a mammogram, it might write a sentence that sounds fluent but gets the medical facts wrong. It might miss a crucial detail or give the wrong risk level.

The Solution: MammoRG (The "Smart Intern")

The authors created a new AI system called MammoRG. Think of MammoRG not as a robot that just guesses words, but as a super-intelligent medical intern who has been trained specifically on how real doctors think.

The system works in two main stages, like a two-step training camp:

Step 1: Learning the "Rulebook" and "Experience" (Cross-Modal Knowledge)

In the first stage, the AI doesn't just look at the picture. It learns to combine the picture with two other things:

The Rulebook (BI-RADS): Doctors use a specific checklist called BI-RADS to decide how risky a finding is (from 1 to 5). The AI learns this rulebook inside out.
The Experience Database: The AI reads thousands of past reports written by real doctors.

The Analogy: Imagine you are trying to write a review of a movie.

Old AI: Just looks at the movie poster and guesses the plot.
MammoRG: Looks at the poster, but also has a notebook of movie reviews written by famous critics and a strict guide on how to rate movies. It uses the poster to find the right page in the notebook and the guide to ensure the rating makes sense.

The paper says this helps the AI understand that two pictures that look similar might actually be very different in terms of risk, something generic AI often misses.

Step 2: Learning the "Special Vocabulary" (Term-Aware Training)

The second stage is about how the AI reads and writes words. Standard AI models break words into tiny, weird pieces (like "calci" and "fication"). This is like trying to understand a sentence by reading it letter-by-letter instead of word-by-word.

The Analogy: Imagine a chef who only knows how to chop ingredients into dust. They can make a soup, but the flavors get mixed up.

MammoRG's Approach: The authors gave the AI a special dictionary where medical terms (like "calcification" or "mass") are treated as whole, unbreakable units.
The Result: The AI can now "taste" the specific medical concepts clearly without them getting muddled. This helps it write reports that use the correct medical terms in the right places.

The New Tool: MammoRGTool (The "Grading Machine")

To prove their new AI was actually good, the authors realized they needed a better way to grade the reports. Standard computer tests just check if the words match (like a spell-checker). But in medicine, getting the facts right is more important than getting the words to rhyme.

So, they built MammoRGTool.

The Analogy: Imagine a teacher grading a student's essay. A spell-checker just counts how many words are spelled right. MammoRGTool is like a strict medical professor who reads the essay and checks: "Did you mention the lump? Did you say where it is? Did you give the correct risk score?"
This tool extracts the specific medical facts from the AI's text and checks them against the truth. The paper claims this tool is incredibly accurate (over 99% accurate on some tests).

The Results: Did It Work?

The authors tested MammoRG on four different sets of data (some from their own hospital, some from other hospitals, and one public dataset).

The Verdict: MammoRG beat all the other top AI models.
The "Why": While other models were good at writing smooth sentences, MammoRG was much better at getting the medical facts right. Specifically, it was significantly better at assigning the correct risk level (BI-RADS score), which is the most critical part of the report for a patient's next steps.

Summary

The paper presents a system that teaches an AI to think like a radiologist by:

Giving it a "rulebook" and "past experience" to consult while looking at images.
Teaching it to speak "medical" fluently by treating complex terms as whole words.
Building a special "grading machine" to ensure the AI gets the medical facts right, not just the grammar.

The result is a computer program that generates mammography reports that are not only well-written but, more importantly, medically accurate and consistent with how real doctors diagnose patients.

Technical Summary: Cross-Modal Clinical Knowledge Integration for Mammography Report Generation

Problem Statement

Breast cancer screening via mammography generates a massive volume of examinations, creating a substantial workload for radiologists. While automated report generation has advanced for general radiology (e.g., chest X-rays, CT), existing methods fail to address the specific complexities of mammography. Current approaches primarily rely on direct visual-to-text mapping, overlooking the structured clinical reasoning process required in practice. Specifically, mammography reporting necessitates:

Multi-view Joint Reasoning: Accurate diagnosis requires integrating information from four standardized views (Right Craniocaudal, Right Mediolateral Oblique, Left Craniocaudal, Left Mediolateral Oblique) and bilateral breast analysis.
Clinical Consistency: Reports must adhere to the Breast Imaging Reporting and Data System (BI-RADS) guidelines, assessing malignancy risk and assigning specific categories that guide clinical decisions.
Terminology Precision: Professional mammography terms are often fragmented by standard tokenizers (e.g., WordPiece, BPE), leading to semantic ambiguity and suboptimal alignment between visual features and diagnostic semantics.

Existing general-purpose models, including large vision-language models (LVLMs), struggle to produce clinically actionable reports, often failing to correctly assign BI-RADS categories or capture fine-grained findings despite high linguistic fluency. Furthermore, the field lacks dedicated tools to evaluate the clinical efficacy of generated reports beyond surface-level text similarity.

Methodology: MammoRG

The authors propose MammoRG, a two-stage training framework designed to simulate the clinical reporting workflow by integrating cross-modal clinical knowledge and employing term-aware fine-tuning.

1. Framework Architecture

MammoRG utilizes an encoder-decoder architecture. The input consists of four standard mammography views, which are encoded by an image encoder ( $E_{img}$ ). The system integrates clinically relevant prior knowledge from a report database and a mammography knowledge graph, fusing these features with image data before feeding them into a text decoder ( $D_{txt}$ ) to generate the final report.

2. Stage 1: Cross-Modal Clinical Knowledge Integration

This stage aims to learn clinically relevant diagnostic semantics under classification-based supervision.

Knowledge Sources: The model leverages a report database (encoded via BERT) and a mammography knowledge graph constructed from a specialized lexicon. The graph encodes semantic relations such as Located_at, Modified_by, and Suggestive_of.
Feature Fusion: The framework employs cross-attention mechanisms at two levels:
- Patient-level: Global image representations attend to report embeddings to capture overall clinical context.
- View-level: Specific view features attend to the knowledge graph embeddings to capture fine-grained diagnostic semantics.
Training Objective: The model is optimized using classification losses (Cross-Entropy for BI-RADS and breast composition; Binary Cross-Entropy for findings and relations) to align visual features with diagnostic semantic contexts.

3. Stage 2: Term-Aware Supervised Fine-Tuning

To address the limitations of standard tokenizers in handling domain-specific terminology, the authors introduce a term-aware tokenizer.

Atomic Semantic Units: Terms from the mammography lexicon are treated as indivisible tokens rather than fragmented subwords. This preserves the atomic semantic integrity of clinical concepts.
Initialization and Tuning: New token embeddings are initialized via a data-driven approach (mean of existing embeddings with perturbations) and optimized during fine-tuning.
Efficiency: Parameter-efficient tuning (LoRA) is applied to the text decoder to adapt the language model to mammography-specific conventions without disrupting pretrained knowledge.

4. MammoRGTool: Clinical Efficacy Evaluation

To facilitate rigorous evaluation, the authors developed MammoRGTool, a dedicated report parsing tool.

Function: It extracts structured clinical information from free-text reports, including BI-RADS categories, tissue composition, abnormal findings, anatomical locations, and relations.
Pipeline: The tool utilizes a rule-based cleaning process and an iterative self-refinement training loop (initially using Qwen3-32B for label extraction) to achieve high-precision extraction. It serves as the ground truth for Clinical Efficacy (CE) metrics.

Key Contributions

MammoRG Framework: A two-stage training framework that explicitly incorporates mammography-specific clinical priors (BI-RADS guidelines, knowledge graphs) and cross-modal semantic alignment, significantly improving both linguistic fluency and diagnostic accuracy.
MammoRGTool: A specialized report parsing tool that achieves a macro-F1 score of over 99% on external validation sets, establishing a reliable benchmark for evaluating the clinical correctness of mammography reports.
Term-Aware Tokenization: A novel approach to handling medical terminology that treats domain-specific terms as atomic units, reducing semantic fragmentation and improving model performance on clinically relevant attributes.
Comprehensive Evaluation: Extensive experiments on internal and external datasets demonstrating state-of-the-art (SOTA) performance on both Natural Language Generation (NLG) and Clinical Efficacy (CE) metrics.

Experimental Results

The model was evaluated on an internal dataset (Yunnan Cancer Hospital) and three external datasets (Guangzhou First People's Hospital, Sun Yat-Sen University, and the public VinDr-Mammo dataset).

Performance: MammoRG consistently outperformed existing methods, including retrained baselines (LLaVA-Rad, R2Gen) and one-shot LVLMs (Qwen3-VL, InternVL3.5).
- Internal Dataset: Achieved the highest scores in all metrics, including BLEU-1 (53.78%), ROUGE-L (62.70%), and critical CE metrics like BI-RADS F1 (26.67%) and Finding F1 (75.09%).
- External Generalization: Demonstrated robust generalization, particularly in BI-RADS assessment. On the VinDr-Mammo dataset, it achieved a BI-RADS F1 of 31.07%, surpassing the second-best model by 4.96%.
Ablation Studies:
- Feature Combination: Integrating image, patient-level, and view-level features yielded the best results, confirming the complementarity of multi-granularity features.
- Component Analysis: Both the term-aware tokenizer and cross-modal knowledge integration contributed significantly to performance gains, with the latter providing the most substantial improvement in clinical consistency.
- Retrieval Comparison: The proposed cross-modal integration outperformed standard image-text and image-based retrieval strategies, indicating that semantic similarity alone is insufficient for capturing clinically critical diagnostic attributes.
Error Analysis: The model showed strong performance on visually salient findings (e.g., masses, ligament thickening) but struggled with subtle findings (e.g., asymmetry, nipple retraction) and fine-grained risk stratification (BI-RADS 4 subcategories), often over-smoothing decisions toward intermediate categories.

Significance and Limitations

Significance:
The paper argues that the clinical value of mammography reports depends more on diagnostic correctness than linguistic fluency. By explicitly modeling the clinical reasoning workflow and integrating structured medical knowledge, MammoRG bridges the gap between generic AI models and clinical utility. The introduction of MammoRGTool addresses a critical gap in the field by providing a standardized method to evaluate clinical efficacy, moving beyond simple text overlap metrics.

Limitations:

Language Dependency: The model is trained exclusively on Chinese mammography reports; its generalizability to other languages and clinical environments is unverified.
Modality Scope: The framework is specific to mammography. Extending it to other imaging modalities would require new domain-specific knowledge representations and annotations.
Fine-Grained Sensitivity: The model exhibits limitations in detecting subtle or underrepresented findings (e.g., asymmetry) and in distinguishing fine-grained BI-RADS subcategories, suggesting a need for improved sensitivity to low-saliency patterns.