Technical Summary: Rad-VLSM

Problem Statement

Medical image segmentation is most clinically valuable when it directly supports diagnosis rather than merely producing lesion masks. However, existing models face significant challenges in real-world clinical workflows, particularly in ultrasound imaging. Lesions are often subtle, localized, and accompanied by weak boundaries, low contrast, and acoustic artifacts (e.g., speckle noise, shadows). Consequently, models relying on global image-level representations often get distracted by background tissues, leading to the dilution of lesion-specific discriminative cues.

Furthermore, while promptable segmentation models like SAM (Segment Anything Model) offer promise, their effectiveness depends heavily on the quality of input prompts. Manual prompts are costly, and automatically generated prompts can be unstable in complex backgrounds. Additionally, simply injecting semantic text descriptions directly into diagnostic classifiers can lead to "shortcut learning," where the model relies on textual correlations rather than grounded visual evidence of lesion morphology and texture. There is a need for a unified framework that connects lesion localization, segmentation, and diagnosis while ensuring the final diagnosis is grounded in spatial and radiomic evidence rather than raw text.

Methodology: Rad-VLSM

The authors propose Rad-VLSM, a two-stage cross-modal framework designed for semantics-assisted lesion focusing, robust segmentation, and visually grounded diagnosis.

Stage I: Language-Guided Lesion Localization and Automated Prompt Generation

This stage utilizes a BLIP-2-based vision-language alignment module to automatically generate candidate bounding box prompts without manual annotation at inference time.

• Asymmetric Prompting Strategy: During training, the model uses detailed BI-RADS morphological descriptions (e.g., shape, margin) to align visual patterns with text. During inference, a single, fixed prompt (e.g., "Breast US: abnormal lesion region") is used universally. This prevents the model from relying on class-specific language shortcuts and forces it to ground predictions in visual evidence.
• Multi-Threshold CAM: The model generates probability heatmaps via Class Activation Mapping (CAM) from text-aligned visual features. Candidate bounding boxes are extracted across multiple thresholds, ranked by confidence, and deduplicated via Intersection over Union (IoU).
• Multi-Candidate Region Aggregation (MCRA): To address the instability of single-box prompts, the authors introduce a three-step aggregation strategy:
  1. Weighted Soft Fusion (V1): Normalizes CAM response scores to fuse masks, logits, and embeddings from multiple candidates.
  2. Semantic-Guided Filtering (V2): Uses cosine similarity between local region embeddings and global text features to re-score candidates, suppressing artifacts (like acoustic shadows) that lack semantic alignment.
  3. Self-Distillation Consistency (V3): A regularization mechanism where high-confidence candidates act as a "teacher" to guide and correct low-confidence "student" predictions, stabilizing training against noisy prompts.

Stage II: SAM-Based Multi-Task Segmentation and Classification

The refined prompts from Stage I are fed into a SAM-based multitask network.

• Segmentation: The SAM image and prompt encoders collaborate to output precise lesion masks.
• Visual-Radiomics Fusion for Diagnosis: Instead of treating diagnosis as an isolated task, the framework uses the predicted masks as spatial priors.
  • Multi-Scale Feature Aggregation: A Squeeze-and-Excitation (SE) mechanism and Adaptive Atrous Spatial Pyramid Pooling (ASPP) capture diverse geometric contexts.
  • Multi-Pooling Classifier: A specialized pooling strategy extracts features from the lesion core, the bounding box neighborhood, and global context, which are concatenated for classification.
  • Dual-Branch Decision Fusion: To enhance interpretability, a parallel Radiomics Head extracts over 500 radiomic features (morphological, first-order, and texture). A cascaded selection strategy (mRMR and LASSO) reduces these to 63 discriminative features. The final diagnosis is a synergistic fusion of the deep visual classification output and the explicit radiomics output.

Key Contributions

1. Rad-VLSM Framework: A semantics-assisted vision-language framework that leverages BLIP-2 for automatic lesion localization and box prompt generation, eliminating the need for manual prompt annotation during inference.
2. MCRA Strategy: A novel multi-candidate region aggregation mechanism that improves the stability of lesion evidence under prompt variation and coarse localization errors, enhancing the robustness of unified diagnosis.
3. Lesion-Grounded Diagnosis: A scheme that integrates implicit visual representations with explicit radiomic features. By using semantic priors only for localization and keeping the final diagnosis centered on lesion-level visual and radiomic evidence, the model avoids semantic shortcut learning.
4. Comprehensive Evaluation: Rigorous evaluation on a private clinical breast ultrasound dataset and multiple public benchmarks (ISIC, BUSI, Colonic Polyp, MRI Brain Tumor), demonstrating strong generalization.

Experimental Results

The framework was evaluated on a private dataset of 559 breast ultrasound images (Cancer vs. Sclerosing Adenosis) and several public benchmarks.

• Segmentation Performance: On the clinical breast ultrasound dataset, Rad-VLSM achieved a mDSC of 0.9217 and mIoU of 0.8579, outperforming state-of-the-art baselines like MedCLIP-SAMv2 (mDSC 0.8865) and BoxShrink (mDSC 0.8735). It also achieved top performance on ISIC 2018 (mDSC 0.9331) and MRI Brain Tumor datasets.
• Diagnostic Performance: In binary classification on the breast ultrasound dataset, Rad-VLSM achieved 94.07% Accuracy and 97.55% AUC, significantly outperforming the strongest baseline (MIAFEx) by 11.77% in accuracy and 9.73% in AUC. It maintained a balanced profile with 98.41% Sensitivity and 89.09% Specificity.
• Ablation Studies:
  • Removing the radiomics branch reduced sensitivity from 98.41% to 88.89%, highlighting its role in mitigating false negatives.
  • The MCRA strategy (V1+V2+V3) was shown to be superior to using a single prompt or ground-truth boxes for downstream diagnosis, as it preserves critical perilesional context.
  • The framework demonstrated robustness to random or mismatched semantic prompts, confirming that the diagnosis relies on visual/radiomic evidence rather than direct text conditioning.
• Interpretability: SHAP analysis revealed that texture-based radiomic features (e.g., wavelet-H glcm Idmn) were the primary drivers of malignant predictions. Cross-correlation analysis showed that deep visual features and radiomic features are complementary rather than redundant.

Significance

The paper claims that Rad-VLSM addresses the critical gap between isolated segmentation and reliable diagnosis in medical imaging. By establishing an evidence-driven pipeline, the framework successfully couples segmentation and diagnosis through lesion-grounded evidence. It reduces dependence on manual prompt annotation, mitigates the impact of imperfect prompt generation through the MCRA strategy, and improves fine-grained lesion characterization by fusing deep learning with radiomics. The authors assert that this approach provides a more transparent and clinically meaningful alternative to existing multi-stage or shared-backbone methods, offering a unified solution that is both highly accurate and interpretable.