MammoWise: Multi-Model Local RAG Pipeline for Mammography Report Generation

MammoWise is a practical, local, multi-model pipeline that leverages open-source Vision Language Models enhanced with few-shot prompting and Retrieval Augmented Generation to generate high-quality mammography reports and perform accurate clinical classifications while ensuring privacy and reproducibility.

The Gist

Imagine you are a radiologist, a doctor who looks at X-ray pictures of breasts to check for cancer. This is a huge job. Every day, they have to look at hundreds of images, spot tiny, subtle clues (like a small lump or a weird pattern), and then write a detailed report for the patient. It's like being a detective who has to solve a mystery and then write a 10-page police report for every single case. It's tiring, time-consuming, and if you make a mistake in the report, it could change a patient's life.

Recently, computers got very smart at looking at pictures and writing words. These are called Vision-Language Models (VLMs). Theoretically, a computer could look at the X-ray and write the report for the doctor. But there are two big problems:

1. Privacy & Cost: The best computers are locked up in giant cloud servers. Sending patient X-rays there is risky and expensive.
2. Reliability: If you just ask a free, open-source computer to write a report, it might "hallucinate" (make things up) or sound like a robot, not a doctor.

Enter MammoWise.

What is MammoWise?

Think of MammoWise not as a single robot doctor, but as a smart workshop or a Swiss Army Knife for building AI assistants. It's a toolkit that lets researchers and doctors take open-source AI models (the free ones) and turn them into reliable, local (on their own computer) report writers.

The authors tested three different "brains" (AI models) inside this workshop:

• MedGemma: A brain trained specifically on medical books.
• LLaVA-Med: A general brain that learned to speak "medical."
• Qwen2.5-VL: A brain good at seeing and understanding complex images.

How Does the Workshop Work?

The paper tests three different ways to teach these AI brains how to do the job, like training a new intern:

1. The "Just Tell Me" Method (Prompting)
You give the AI a picture and a note saying, "You are a doctor. Look at this and write a report."

• The Result: It's okay, but sometimes the AI gets confused or misses details. It's like asking a smart student to take a test without any study guides.

2. The "Show Me Examples" Method (RAG - Retrieval Augmented Generation)
This is the paper's secret sauce. Before the AI looks at the new patient, MammoWise searches a database of thousands of past, correct reports. It finds 5 cases that look very similar to the current one and says, "Hey AI, look at these past reports first. They are similar to this one. Now, write your report."

• The Analogy: It's like giving the intern a stack of "cheat sheets" from previous successful cases right before they start writing.
• The Result: The reports sound much more professional and accurate. The AI stops guessing and starts mimicking real doctors.

3. The "Specialized Training" Method (Fine-Tuning)
Sometimes, just showing examples isn't enough for the tricky parts (like deciding if a lump is definitely cancer or just a cyst). So, the authors took the best AI brain (MedGemma) and gave it a crash course. They fed it thousands of labeled examples until it learned the specific rules of breast cancer detection.

• The Analogy: This is like taking that smart intern and sending them to a 3-month intensive medical residency.
• The Result: This was the game-changer. The AI became incredibly accurate at spotting specific details, almost as good as the best specialized software in the world, but it runs on a regular computer in the hospital, not a supercomputer in the cloud.

The Big Takeaways

The researchers found a perfect "division of labor" for using these tools:

• For Writing the Report: You don't need to train the AI heavily. Just giving it a few examples (RAG) is enough to make it write beautiful, professional-sounding reports that doctors can use as a draft.
• For Making the Diagnosis (The Labels): If you need the AI to be 100% sure about a specific label (like "BI-RADS 4" or "Calcification present"), you do need the specialized training (Fine-Tuning).

Why Does This Matter?

MammoWise proves that we don't need to send our private medical data to big tech companies in the cloud. We can build our own, secure, private AI assistants right in the hospital.

It's like moving from renting a expensive, private jet (cloud AI) to building a reliable, custom car (local AI) that you can drive anywhere, keep your secrets in, and tune up exactly how you want. This makes advanced medical AI accessible, safe, and ready to help doctors save time and catch cancer earlier.

Technical Summary

1. Problem Statement

Screening mammography is a high-volume, time-sensitive clinical task requiring radiologists to translate subtle visual findings into standardized reports (BI-RADS assessments, breast density categories, and narrative findings). While Vision-Language Models (VLMs) offer potential for automation, current solutions face significant barriers:

• Privacy and Cost: State-of-the-art demonstrations often rely on closed-source, cloud-hosted models, raising concerns regarding Protected Health Information (PHI) transmission, operational costs, and reproducibility.
• Lack of Adaptability: Existing open-source VLMs are often not specialized for mammography. They struggle with "out-of-the-box" performance, producing clinically implausible details or hallucinations without careful guidance.
• Fragmented Workflows: Researchers and practitioners lack a unified framework to compare different open VLMs, prompting strategies, and adaptation techniques (prompting vs. fine-tuning) within a local, reproducible environment.

2. Methodology: The MammoWise Pipeline

The authors propose MammoWise, a modular, local, multi-model pipeline designed to turn open-source VLMs into clinically styled report generators and multi-task classifiers. The system is built on Ollama for local model hosting and supports three primary adaptation strategies:

A. Data Collection & Preprocessing

• Datasets: The study utilizes VinDr-Mammo (20,000 images from 5,000 patients) and DMID (510 lesion-enriched images).
• Image Processing: For VinDr-Mammo, four views per patient (Right/Left CC and MLO) are merged into a single composite image (2x2 grid) to mimic radiologist workflow.
• Label Construction: Reports are synthesized into JSON format containing Breast Density (ACR A-D), BI-RADS (1-5), Findings (mass, calcification, etc.), and a derived "Suspicion" label.
• RAG Preparation: Image-text pairs are converted into multimodal embeddings using OpenCLIPEmbedding and stored in ChromaDB (a vector database) for retrieval.

B. Model Selection

Three open-source VLMs hosted locally via Ollama were evaluated:

1. MedGemma (4B): A medical-specific model derived from Google's Gemma family, trained on radiology/pathology data.
2. LLaVA-Med (7B): A biomedical fine-tuned version of LLaVA.
3. Qwen2.5-VL (7B): A general-purpose multimodal model with strong reasoning capabilities.

C. Adaptation Strategies

The pipeline evaluates three distinct approaches to improve performance:

1. Prompting:
   • Zero-shot: High-level role instructions and output format constraints.
   • Few-shot: Adding fixed examples of image-report pairs.
   • Chain-of-Thought (CoT): Step-by-step reasoning prompts guiding the model through density assessment, finding identification, and BI-RADS assignment.
2. Retrieval-Augmented Generation (RAG):
   • A multimodal retrieval system dynamically selects the top 5 most similar image-report pairs from the training set (using cosine similarity) and injects them into the prompt as context. This aims to ground the model in specific clinical examples.
3. Parameter-Efficient Fine-Tuning (PEFT):
   • QLoRA (Quantized Low-Rank Adaptation) was applied to MedGemma on the VinDr-Mammo dataset.
   • Class Balancing: Due to severe class imbalance (e.g., few BI-RADS 5 cases), the training set was rebalanced via downsampling majority classes and augmenting minority classes (flipping, scaling, translation).
   • Training Modes: Evaluated both Multi-task generation (full JSON report in one go) and Single-task generation (predicting one field at a time).

D. Evaluation Metrics

• Classification: Macro-averaged Accuracy, Precision, Recall, F1-score, and Specificity for BI-RADS, Breast Density, and binary findings (Mass, Calcification, Asymmetry, Suspicion).
• Generation: BERTScore and ROUGE-L to measure semantic and lexical similarity between generated reports and ground truth.

3. Key Contributions

1. Unified Local Pipeline: Introduction of MammoWise, a reusable scaffold that supports any Ollama-hosted VLM, enabling local, privacy-preserving experimentation without cloud dependency.
2. Multimodal RAG Workflow: A specific implementation of RAG for mammography that retrieves image-text exemplars to provide case-specific context, addressing hallucination risks.
3. Systematic Comparison: A comprehensive evaluation across three models, two datasets, and three adaptation strategies (Prompting, RAG, QLoRA), providing a clear hierarchy of performance.
4. Evidence for PEFT: Demonstration that while prompting is sufficient for drafting reports, lightweight fine-tuning (QLoRA) is critical for stabilizing high-fidelity structured classification.

4. Key Results

The study evaluated the models on VinDr-Mammo and DMID, yielding the following insights:

• Report Generation:
  • Local VLMs generate high-quality narrative reports even with zero-shot or few-shot prompting.
  • RAG Impact: RAG generally improves narrative similarity (BERTScore/ROUGE-L) and helps anchor phrasing. However, results were dataset-dependent; on DMID, RAG sometimes introduced noise, reducing similarity in "Findings" text due to mismatched retrieval contexts.
• Structured Classification (Prompting vs. RAG):
  • Prompting-only classification was highly variable and often poor (e.g., BI-RADS accuracy ~0.29 for best prompting on VinDr-Mammo).
  • RAG provided modest, incremental gains (e.g., BI-RADS accuracy improved to ~0.32), but was insufficient for reliable clinical decision-making.
• Impact of QLoRA Fine-Tuning:
  • Fine-tuning yielded substantial improvements over both prompting and RAG.
  • MedGemma (Single-Task QLoRA) achieved:
    • BI-RADS Accuracy: 0.7545 (vs. ~0.32 with RAG).
    • Breast Density Accuracy: 0.8840.
    • Calcification Accuracy: 0.9341.
    • Mass Accuracy: 0.8740.
  • The study found that optimal performance varied by task and checkpoint (e.g., 10 epochs for BI-RADS, 6 epochs for Mass), suggesting the need for early stopping or checkpoint selection.
• Comparison to SOTA: The fine-tuned MedGemma model outperformed several specialized SOTA models (e.g., PubMedCLIP, LLaVA-Mammo) on BI-RADS and Breast Density tasks on the same datasets.

5. Significance and Conclusion

MammoWise demonstrates that a pragmatic, local approach to mammography AI is viable and effective. The paper establishes a "division of labor" for medical VLM deployment:

1. Prompting + RAG: Sufficient for drafting clinically styled narrative reports and providing context.
2. Lightweight Fine-Tuning (QLoRA): Essential for achieving the high reliability required for structured clinical labels (BI-RADS, density, specific findings).

Limitations: The study was limited to two datasets (potential domain shift), did not explore advanced prompting (e.g., Tree-of-Thought), and relied on a rebalanced training subset which may affect calibration in real-world prevalence scenarios.

Conclusion: The authors provide a practical blueprint for deploying local, privacy-preserving VLMs in breast cancer screening. By decoupling the pipeline from specific models and integrating prompting, RAG, and fine-tuning, MammoWise offers a reproducible framework for future research and clinical prototyping. The code and pipeline are open-sourced on GitHub.