Technical Summary: MIFR – A Modality-Invariant and Fair Representation Framework for Skin Disease Classification

1. Problem Statement

Skin disease diagnosis via deep learning faces two critical barriers to real-world clinical deployment:

1. Modality Heterogeneity: Skin lesions are captured via heterogeneous imaging modalities, primarily clinical photographs and dermoscopic images. Existing models often rely on a single modality, which can be insufficient for accurate diagnosis.
2. Demographic Disparities: Public dermatology datasets are dominated by lighter skin tones (Fitzpatrick Skin Types I–III), causing models to perform significantly worse on darker skin (FST IV–VI). This perpetuates healthcare disparities.

Current approaches typically address these issues in isolation. Fairness-aware methods (e.g., FairDisCo, PatchAlign) generally operate on single modalities, while multimodal foundation models (e.g., PanDerm) often lack explicit fairness constraints. Furthermore, many fairness methods require sensitive attribute labels (skin tone) at inference time, which is impractical in clinical settings.

2. Methodology: The MIFR Framework

The authors propose MIFR (Modality-Invariant and Fair Representation), a dual-encoder architecture designed to jointly process clinical and dermoscopic images while enforcing fairness without requiring sensitive attributes during inference.

2.1 Architecture

• Dual Encoders: The model employs two independent Vision Transformer (ViT-Small) encoders: one for clinical images ($f^c_\theta$) and one for dermoscopic images ($f^d_\phi$).
• Projection Heads: Each encoder output is passed through a modality-specific projection head (a multi-layer perceptron with batch normalization and GELU activations) to map inputs into a shared high-dimensional embedding space ($\mathbb{R}^{512}$).
• Inference Flexibility: At inference, the auxiliary branches are discarded. The model can operate on clinical images, dermoscopic images, or both, eliminating the need for separate specialized models.

2.2 Multi-Objective Loss Function

The training process optimizes a unified loss function ($L_{Total}$) comprising five components:

1. Target Classification ($L_{cls}$): A weighted cross-entropy loss addressing class imbalance to ensure accurate disease prediction ($y_i$).
2. Skin-Tone Attribute Branch (SA): An adversarial mechanism to remove skin-tone bias ($s_i$) from features without needing $s_i$ at inference.
   • Confusion Loss ($L_{conf}$): Trains the main encoder to make the skin-tone classifier uncertain (maximizing entropy), effectively hiding skin-tone cues.
   • Skin Loss ($L_{skin}$): Trains the auxiliary skin-tone classifier to predict the true label, preventing the encoder from collapsing into a trivial solution.
3. Contrastive Branch ($L_{contr}$): A supervised contrastive loss applied per modality to pull embeddings of the same disease class together and push different classes apart, establishing intra-class compactness.
4. Modality-Invariant Branch ($L_{MI}$): Based on VICReg, this enforces alignment between clinical and dermoscopic embeddings of the same lesion. It includes:
   • Invariance ($L_{inv}$): Minimizes cosine distance between paired embeddings.
   • Variance ($L_{var}$): Prevents dimensional collapse by ensuring sufficient variance in each embedding dimension.
   • Covariance ($L_{cov}$): Decorrelates embedding dimensions to reduce redundancy.

3. Experimental Setup

• Datasets:
  • Internal Training: Paired HIBA (1,195 images with FST) and Derm7pt (1,738 images with estimated FST) datasets, filtered for Melanoma (MEL), Nevus (NV), and Basal Cell Carcinoma (BCC).
  • External Benchmarks: PAD-UFES-20 (clinical images) and ISIC 2019 (dermoscopic images).
• Baselines: Comparison against FairDisCo and PatchAlign.
• Metrics: Accuracy (Acc), F1-score, AUC, Predictive Quality Disparity (PQD), and Accuracy Gap ($\Delta$Acc).

4. Key Results

4.1 Internal Performance (HIBA + Derm7pt)

• Fairness: The full MIFR model achieved a PQD of 87.23%, substantially outperforming FairDisCo (66.67%) and PatchAlign (82.98%).
• Accuracy: MIFR achieved 93.33% accuracy and 92.17% F1-score. While slightly lower than the baselines (93.81%), the performance remains highly competitive.
• Ablation: The combination of contrastive learning and modality invariance (BASE+Contr+MI) yielded the lowest accuracy gap (0.0265), suggesting that modality invariance implicitly promotes fairness.

4.2 External Generalization

• PAD-UFES-20 (Clinical): MIFR significantly outperformed baselines in Accuracy (85.10% vs. 78.35% for FairDisCo) and AUC (91.89% vs. 83.44%).
• ISIC 2019 (Dermoscopic): MIFR achieved 76.64% accuracy, matching FairDisCo and surpassing PatchAlign. However, the AUC (80.50%) was lower than the base model (86.62%). The authors attribute this to the trade-off where enforcing modality invariance sacrifices some modality-specific discrimination for cross-modality generalizability.

4.3 Visualization

t-SNE visualizations confirmed that:

1. Clinical and dermoscopic embeddings of the same disease are geometrically aligned (modality invariance).
2. Skin tones are near-perfectly mixed within disease clusters, indicating the removal of skin-tone bias (fairness).

5. Significance and Claims

The paper claims that MIFR represents a significant step toward equitable and robust skin disease diagnosis by:

• Unifying Objectives: Successfully treating modality invariance and fairness as a joint optimization problem rather than separate challenges.
• Practical Deployment: Eliminating the need for sensitive attribute labels at inference time, a requirement that limits many existing fairness-aware methods.
• Cross-Dataset Robustness: Demonstrating superior generalization on external benchmarks (PAD-UFES-20), validating its applicability in real-world scenarios where data distributions shift.

6. Limitations and Future Directions

The authors acknowledge specific constraints:

• Data Dependency: The framework relies on paired training data (clinical and dermoscopic images of the same lesion), which are often scarce.
• Computational Cost: The dual-encoder architecture increases computational overhead compared to single-encoder models.
• Dataset Bias: The study utilized international repositories lacking African skin lesion data. The authors note that public datasets remain skewed toward lighter skin tones, and adapting MIFR to African clinical datasets (e.g., eSkinHealth) is a necessary future direction to address the specific underperformance of prior methods on FST IV–VI.
• Hyperparameter Tuning: The interaction between the five loss weights required manual tuning on the validation set rather than an exhaustive search, which may not guarantee optimal performance.