Multimodal Skin Lesion Classification with Swin Transformer and Clinical Metadata Fusion

Technical Summary: Multimodal Skin Lesion Classification with Swin Transformer and Clinical Metadata Fusion

Problem Statement
Automated skin lesion classification is critical for the early diagnosis of skin cancer but faces significant challenges in clinical deployment. These challenges include severe class imbalance in datasets, high inter-class similarity between lesion types, and substantial intra-class variability due to differences in pigmentation, anatomical site, and lesion appearance. Furthermore, existing deep learning approaches often suffer from dataset-specific visual biases, poor sensitivity to minority classes (such as rare malignant categories), and a lack of reliability metrics like calibrated confidence and uncertainty estimation. The reliance on image data alone often fails to capture the full clinical context, while standard evaluation methods may suffer from data leakage if patient-level correlations are not properly managed.

Methodology
The authors propose a multimodal classification framework that integrates dermoscopic image features with structured clinical metadata (patient age, sex, and anatomical site) to enhance diagnostic performance.

• Dataset and Preprocessing: The study utilizes the HAM10000 dataset, comprising 10,015 images across seven diagnostic classes. To address the inherent class imbalance (where benign nevi constitute ~67% of the data), the authors employ a group-aware data partitioning strategy. Samples are split by patient or lesion identifier to prevent data leakage between training and test sets. Preprocessing involves resizing images to 224×224 pixels, applying data augmentation (flipping, rotation, color jittering, random erasing) during training, and normalizing clinical metadata (scaling age, one-hot encoding sex and site).
• Architecture: The core of the model is a dual-branch network:
  1. Image Branch: Utilizes a Swin Transformer backbone to extract hierarchical, attention-based visual features from dermoscopic images. The final classification layer of the backbone is removed to treat the output as a global descriptor.
  2. Metadata Branch: Processes structured clinical data through a multilayer perceptron (MLP) to generate a compact feature embedding.
  3. Fusion: The image and metadata embeddings are concatenated and passed through a lightweight classification head to produce class probabilities.
• Training Strategy: The model is trained using class-weighted cross-entropy loss to mitigate the bias toward majority classes. Optimization is performed using an adaptive gradient-based optimizer with a cosine learning-rate schedule and mixed-precision computation.
• Reliability and Explainability: To ensure clinical trustworthiness, the authors apply temperature scaling as a post-hoc calibration method to align predicted probabilities with observed accuracy. Additionally, uncertainty estimation and Explainable AI (XAI) techniques (attribution heatmaps and attention maps) are employed to verify that the model focuses on lesion regions rather than background artifacts.

Key Contributions

1. Multimodal Framework: Development of a collaborative visual-context learning system that fuses Swin Transformer-based image representations with structured clinical metadata.
2. Reliability-Oriented Modeling: Integration of calibrated probability estimation and predictive uncertainty analysis to improve confidence in automated diagnoses, addressing the gap in trust for medical AI.
3. Interpretability: Adaptation of XAI techniques to the multimodal transformer architecture to provide clinically interpretable visual evidence, demonstrating that the model attends to relevant lesion structures.
4. Rigorous Evaluation: Implementation of leakage-aware evaluation via group-based splitting and comprehensive performance analysis across imbalanced classes.

Experimental Results
Evaluated on the HAM10000 test set, the proposed multimodal model achieved a test accuracy of 92.55% and a macro F1-score of 91.33%.

• Class Performance: The model demonstrated strong performance across minority classes, achieving high recall for rare categories such as dermatofibroma (df: 90.91%), vascular lesions (vasc: 94.12%), and actinic keratoses (akiec: 91.18%).
• Comparison: The multimodal approach outperformed unimodal baselines, where the image-only model achieved 88.62% accuracy and the metadata-only model achieved only 11.13% accuracy. The proposed method also showed competitive or superior results compared to other studies on the same dataset (e.g., CNN-LSTM fusion at 92.0%).
• Calibration: Temperature scaling significantly improved model reliability, reducing the Expected Calibration Error (ECE) from 5.18% (uncalibrated) to 0.91%. The calibrated model showed high average confidence (0.84–0.96) with low Negative Log-Likelihood (NLL) and Brier scores.
• Qualitative Analysis: XAI visualizations confirmed that the model's attention mechanisms focused on lesion-specific structures rather than background noise, supporting the clinical plausibility of the predictions.

Significance and Claims
The paper claims that multimodal fusion, when combined with calibration and interpretability analysis, provides an effective and trustworthy approach for automated skin lesion classification. The study emphasizes that integrating clinical metadata with advanced visual transformers not only improves diagnostic accuracy, particularly for underrepresented classes, but also enhances the reliability of the system through calibrated confidence estimates. By demonstrating that the model focuses on clinically relevant regions and provides well-calibrated probabilities, the authors argue that this framework addresses key limitations in current medical AI, such as poor minority-class sensitivity and lack of transparency, thereby offering a more robust foundation for clinical decision support. Future work is directed toward external validation, the inclusion of richer clinical attributes, and the development of uncertainty-aware referral mechanisms.