💻 computer science

Boot-and-Feedback Framework for Generalist-Expert Model Collaboration in Breast Ultrasound Diagnosis

The paper proposes the Boot-and-Feedback (BooF) framework, which synergistically combines Multimodal Large Language Models and vision-expert models through a lexicon-guided bootstrapping stage and an attention-based feedback fusion stage to significantly improve the diagnostic accuracy and interpretability of breast ultrasound analysis while mitigating hallucinations.

Original authors: Ming Cheng, Hongyu Sun, Zhaolin Chen, Jun Liu, Hossein Rahmani, Qiuhong Ke

Published 2026-08-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ming Cheng, Hongyu Sun, Zhaolin Chen, Jun Liu, Hossein Rahmani, Qiuhong Ke

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Breast cancer remains the most common malignancy affecting women worldwide, making early and accurate detection a matter of life and death. For decades, doctors have relied on breast ultrasound, a safe and accessible imaging technique, to spot suspicious lumps. However, interpreting these images is a skill that varies from one operator to another; what one radiologist sees as a harmless cyst, another might flag as a potential tumor. To help bridge this gap, scientists have long tried to build computer systems that can read these scans with the same consistency as a human expert. While modern artificial intelligence has made great strides in spotting patterns, a new challenge has emerged: the rise of powerful language models that can describe images in human words. These models are excellent at general tasks but often struggle with the strict rules of medicine, sometimes inventing details that aren't there or misinterpreting what they see. This creates a dangerous situation where a computer might confidently describe a benign lump as malignant, leading to unnecessary panic or, worse, missed diagnoses.

A team of researchers from Monash University, Renmin University of China, and Lancaster University has developed a new way to solve this problem, creating a system where two different types of artificial intelligence work together in a loop rather than just passing a message one way. Their approach, which they call the Boot-and-Feedback framework, treats the diagnosis not as a single step, but as a conversation between a generalist and a specialist. The generalist is a large language model capable of generating detailed descriptions, while the specialist is a vision expert trained specifically to recognize cancer patterns in ultrasound scans. The researchers found that simply letting the language model describe the image and then handing that description to the specialist often fails; the language model's errors can confuse the specialist, lowering the overall accuracy. Instead, they designed a process where the specialist first gives a preliminary opinion to the language model, which then uses that hint to write a much more accurate, medically grounded description. This description is then fed back to the specialist, who uses it to make a final, highly reliable diagnosis.

The process begins with a "boot" stage, where the system prepares the language model to think like a doctor. The researchers realized that asking a general AI to guess if a tumor is cancerous leads to wild guesses. To fix this, they forced the model to speak only in the standard language doctors use, known as BI-RADS. This system breaks down a breast lesion into five specific, observable traits: its shape, its orientation, the clarity of its edges, its internal texture, and how sound waves behave behind it. By restricting the language model to describe only these five features, the system prevents it from making up a diagnosis. But the researchers went a step further. They also gave the language model a "hint" from the vision expert—a preliminary guess about whether the lump is benign or malignant. The language model is instructed to use this hint as a reference point but to remain independent; if the image clearly contradicts the hint, the model is told to describe what it actually sees. This two-part guidance ensures the language model produces a description that is both medically standard and consistent with the visual evidence, effectively stopping it from hallucinating details.

Once the language model has generated this high-quality, structured description, the system moves to the "feedback" stage. Here, the description is combined with the original ultrasound image and fed into the final expert model. The researchers designed a special mechanism that acts like a filter, allowing the expert to weigh the importance of the text against the image. If the text describes a feature that doesn't match the visual data, the system learns to ignore that part of the text and trust the image. This prevents the final diagnosis from being misled by any remaining errors in the description. The result is a system that benefits from the language model's ability to articulate complex medical concepts and the vision model's ability to see the raw data, with each step correcting the weaknesses of the other.

When the team tested this framework on two large public datasets of breast ultrasound images, the results were clear. The new system significantly outperformed existing methods in both accuracy and the ability to correctly identify cancerous tumors. On one dataset, the system achieved an accuracy of 93.4 percent, a notable improvement over previous state-of-the-art models. More importantly, the system did not just get the right answer; it provided a clear, step-by-step explanation of why it reached that conclusion, using the standard medical terms that doctors rely on. This combination of high precision and clear reasoning suggests that such a framework could become a valuable tool for radiologists, offering a second opinion that is both reliable and easy to understand. The study demonstrates that by guiding artificial intelligence with structured medical knowledge and allowing it to learn from the mistakes of its partners, we can build systems that are not only smarter but also safer for clinical use.

Technical Summary: Boot-and-Feedback Framework for Generalist-Expert Model Collaboration in Breast Ultrasound Diagnosis

Problem Statement
Breast ultrasound (BUS) is a critical tool for breast cancer diagnosis but remains heavily operator-dependent, leading to variability in clinical settings. While deep learning models (CNNs, ViTs) have improved diagnostic accuracy, they often lack structured clinical reasoning and interpretability. Conversely, Multimodal Large Language Models (MLLMs) can generate human-readable diagnostic descriptions but suffer from "hallucinations" and domain misalignment due to a lack of medical priors. Recent attempts to simply couple MLLMs with expert models often fail; preliminary experiments indicate that unidirectional interactions where MLLMs generate descriptions without constraints can mislead expert models, causing performance degradation and conflicting predictions.

Methodology: The Boot-and-Feedback (BooF) Framework
The authors propose the Boot-and-Feedback (BooF) framework, a collaborative architecture designed to enable bidirectional interaction between a generalist MLLM and a domain-specific Vision-Language Expert. The framework operates in two distinct stages:

Boot Stage (MLLM Domain Adaptation):
This stage adapts the MLLM to the BUS domain to prevent hallucinations and ensure clinical consistency before it interacts with the expert model. It employs two specific strategies:
- BI-RADS Lexicon Alignment: Instead of asking the MLLM for direct benign/malignant predictions, the system prompts the MLLM (role-playing as a radiologist) to generate descriptions based on the standardized BI-RADS lexicon. The model is constrained to describe five specific lesion properties: shape, orientation, margin, internal echogenicity, and posterior features. This bridges general visual competence with domain-specific terminology.
- Vision Expert Prediction Priors Guidance (VEPP): To further ground the MLLM, a lightweight vision expert model provides a preliminary probabilistic prediction (benign vs. malignant). This prediction is converted into a structured prior $P = \{hypothesis, confidence\}$ . This prior is injected into the MLLM's prompt as a reference. Crucially, the MLLM is instructed to use this prior as a guide but to explicitly flag contradictions if the image evidence disagrees. This narrows the reasoning space while allowing the MLLM to challenge erroneous priors, reducing hallucinations without forcing blind adherence.
Feedback Stage (Multimodal Expert Enhancement):
The structured, BI-RADS-aligned descriptions generated by the MLLM are fed into a downstream Vision-Language Expert alongside the original ultrasound image.
- Architecture: The system uses a frozen text encoder (to preserve standardized semantics) for the BI-RADS attributes and a trainable image encoder for the BUS image.
- Attention-Gated Cross-Modality Fusion Module (AGCFM): To address residual noise or potential hallucinations in the text, the framework employs a lightweight attention mechanism. This module computes cross-attention between the image features (Query) and the text embeddings (Keys). It adaptively weights the textual descriptors, allowing the expert model to leverage relevant semantic priors while suppressing irrelevant or hallucinated noise. The fused representation is then used for the final malignancy classification.

Key Contributions

Bidirectional Collaboration: Unlike previous unidirectional approaches where MLLMs simply feed text to experts, BooF establishes a feedback loop where the expert's visual priors guide the MLLM (Boot), and the MLLM's structured text refines the expert's visual features (Feedback).
Hallucination Mitigation: By constraining MLLM output to the BI-RADS lexicon and grounding it with vision expert priors, the framework significantly reduces spurious descriptions that typically compromise clinical validity.
Adaptive Fusion: The introduction of the Attention-Gated Cross-Modality Fusion Module allows the system to dynamically filter semantic noise, ensuring that only reliable textual feedback influences the final diagnosis.

Experimental Results
The framework was evaluated on two public datasets: BUS-BRA and BUSI.

Ablation Studies: On the BUS-BRA dataset, the authors demonstrated that coupling an MLLM without bootstrapping strategies could actually degrade performance (e.g., AUC dropping below vision-only baselines).
- Adding BI-RADS Lexicon Alignment provided consistent gains.
- Adding VEPP yielded the most substantial improvements, particularly in Recall and AUC.
- The full BooF configuration (including the Attention-Gated module) achieved the best results across all metrics. Across both backbones, BooF improved AUC by approximately 4.8%–6.1% over the vision-only baseline (specifically 6.1% for ResNet-50 and 4.8% for MedViT), with accuracy improvements of 5.7%–7.4% and recall improvements of 14%–16%.
State-of-the-Art Comparison: BooF outperformed existing methods (including KRC-APM, HoverTrans, REAF, BD-StableNet, and SgmaFuse) on both datasets. On BUSI, it achieved an AUC of 0.959 and Specificity of 0.962. On BUS-BRA, it reached an AUC of 0.976 and F1-score of 0.899.
Interpretability: Beyond accuracy, the framework provides BI-RADS-aligned descriptions, offering clinically grounded rationales that facilitate radiologist verification.

Significance and Claims
The paper claims that BooF successfully addresses the dual challenges of diagnostic reliability and interpretability in breast ultrasound. By fostering a synergistic interaction between generalist reasoning (MLLM) and domain-specific expertise (Vision Expert), the framework ensures that diagnostic predictions are both accurate and clinically aligned. The authors assert that their approach is highly adaptable to other specialized medical classification tasks and that the boot-and-feedback paradigm offers a robust solution to the limitations of current unidirectional multimodal systems. Future work is suggested to explore multi-round iterative interactions to further enhance robustness.

Technical Summary: Boot-and-Feedback Framework for Generalist-Expert Model Collaboration in Breast Ultrasound Diagnosis

More like this