A Hybrid CNN-Transformer Deep Learning Model for Differentiating Benign and Malignant Breast Tumors Using Multi-View Ultrasound Images

Technical Summary: A Hybrid CNN-Transformer Deep Learning Model for Breast Tumor Differentiation

Problem Statement
Breast cancer remains a leading global malignancy, necessitating early and accurate diagnosis. While ultrasound is a pivotal, non-invasive screening tool, its diagnostic accuracy is heavily dependent on operator experience and often relies on single-image analysis. This approach fails to capture the comprehensive, multi-dimensional features of a lesion that clinicians synthesize by scanning from multiple angles and planes. Furthermore, existing deep learning studies frequently treat individual 2D ultrasound images as independent samples, diverging from clinical workflows where diagnosis relies on cumulative evidence from a sequence of images. Additionally, standard sequential models like RNNs are ill-suited for the unordered sets of images typical in ultrasound examinations, and random data partitioning often leads to data leakage, artificially inflating performance metrics.

Methodology
The study proposes a novel Computer-Aided Diagnosis (CAD) system that emulates a clinician's multi-view diagnostic process using a hybrid deep learning architecture.

• Data Strategy: The study utilized a retrospective dataset from two hospitals (840 patients internal, 133 external) and a prospective validation set (188 new lesions). A rigorous lesion-level data partitioning strategy was employed, ensuring all images from a single tumor were assigned exclusively to either the training, validation, or test set to prevent data leakage.
• Preprocessing: Raw DICOM images were cropped to isolate lesions, resized to 300×300 pixels, and normalized. To handle variable image counts per lesion (2–10 images), the authors implemented a fixed-length strategy (max 10 images) using zero-padding and an attention mask to ensure padded dummy images did not influence the model.
• Model Architecture:
  • Backbone (Local Feature Extraction): A pre-trained EfficientNetV2 model extracts spatial features from each individual ultrasound image independently.
  • Fusion (Global Feature Integration): The extracted feature vectors are fed into a Transformer Encoder. Crucially, positional encoding was removed to enforce permutation invariance, treating the input as an unordered set rather than a sequence. This aligns with the clinical reality that the acquisition order of ultrasound views is arbitrary.
  • Classification: A learnable classification token aggregates information from all views via self-attention, followed by a Multi-Layer Perceptron (MLP) head with a Sigmoid activation to predict malignancy probability.
• Comparative Baselines: The proposed model was benchmarked against:
  • A single-image EfficientNetV2 classifier.
  • A CNN-only model using global average pooling (bag-of-features).
  • A CNN-RNN (LSTM) model to test sequential dependency hypotheses.
  • State-of-the-art architectures: VGG-ViT, TransMed, and MedFormer.

Key Contributions

1. Novel Hybrid Architecture: Development of an end-to-end model integrating EfficientNetV2 with a Transformer, specifically designed to treat multi-view ultrasound images as an unordered set.
2. Methodological Rigor: Implementation of strict lesion-level data partitioning to eliminate data leakage and ensure generalizable performance evaluation.
3. Clinical Alignment: The model's design (permutation invariance) mirrors the clinical cognitive process of synthesizing information from multiple views without relying on arbitrary acquisition order.
4. Comprehensive Validation: The study includes internal testing, external validation across different vendors (GE and Philips), and a prospective validation on a newly collected, independent dataset.

Results

• Performance: On the internal test set, the hybrid model achieved an Accuracy of 0.960 and an AUC of 0.9788. On the external dataset, it maintained robust performance with an Accuracy of 0.940 and an AUC of 0.9730.
• Prospective Validation: In a prospective test on 188 new lesions, the model achieved an Accuracy of 0.952 and an AUC of 0.9801, confirming stability on temporally separated real-world data.
• Comparative Superiority: The hybrid model significantly outperformed the single-image baseline (AUC 0.95/0.94) and other architectures (VGG-ViT, TransMed, MedFormer). The ablation study confirmed that the Transformer's self-attention mechanism outperformed both simple averaging (CNN-Only) and sequential modeling (CNN-RNN), validating the hypothesis that unordered set modeling is superior for this task.
• Generalization: The model demonstrated consistent performance across different ultrasound machine vendors (GE LOGIQ E9 vs. Philips EPIQ 7C), indicating it learns biologically relevant features rather than device-specific artifacts.
• Expert Comparison: In a blind test, the model (Accuracy 0.960, Sensitivity 0.967) significantly outperformed both junior and senior human radiologists (Senior Expert Accuracy 0.849, Sensitivity 0.869).
• Efficiency: The model demonstrated ultra-low latency, averaging 0.0335 seconds per patient, making it suitable for real-time clinical integration.

Significance and Claims
The paper claims that combining CNNs with Transformers yields a highly accurate and robust diagnostic system that aligns with clinical logic. By effectively fusing multi-view information, the model addresses the limitations of single-image analysis and operator dependency. The authors assert that their approach offers a computationally effective solution to the core challenge of multi-view integration, demonstrating that a task-specific, lightweight architecture can outperform complex, general-purpose medical transformers. The study concludes that this system has the potential to improve diagnostic reliability, reduce false negatives (missed malignancies), and decrease unnecessary biopsies by improving specificity, serving as a reliable assistive tool for clinicians.