Effect of Demographic Bias on Skin Lesion Classification

Technical Summary: Effect of Demographic Bias on Skin Lesion Classification

Problem Statement
Deep learning models for medical imaging, while achieving high overall diagnostic performance, often exhibit biases against specific demographic groups. In skin lesion classification, performance disparities can arise from representation bias—systematic imbalances in training data regarding protected attributes such as patient sex and age. Previous studies have often examined these factors in isolation or within single imaging modalities without systematic control over data distributions. Furthermore, the relative effectiveness of different bias mitigation strategies (single-task, multi-task, and adversarial learning) across extreme distributional imbalances remains under-explored. This study addresses the need for a comprehensive evaluation of how controlled demographic skews in training data influence model performance and whether specific learning strategies can effectively mitigate these biases.

Methodology
The authors employed a rigorous experimental pipeline using three datasets: a curated subset of the ISIC archive (dermoscopic images) for training and internal validation, and two external datasets, PAD-UFES-20 (smartphone images) and DERM7PT (dermoscopic images), for cross-dataset validation.

• Dataset Construction via Linear Programming (LP): To ensure precise control over demographic distributions, the authors utilized a linear programming method rather than random sampling. This allowed them to generate training and validation datasets with specific, pre-defined ratios of sex (ranging from 100% male to 100% female) and age (spanning five age brackets: 0–50, 51–60, 61–70, 71–80, and 81+). They created seven distinct sex-distribution scenarios and three age-distribution schemes (Younger-skewed, Balanced, Older-skewed).
• Model Architectures: Three ResNet-50-based architectures were evaluated:
  1. Single-Task Baseline: A standard binary classification model for malignant vs. benign lesions.
  2. Reinforcing Multi-Task Model: A model with an auxiliary head that predicts the demographic attribute (sex or age) alongside the primary classification task. The authors note "reinforcing" refers to strengthening the influence of the demographic signal to regularize the shared encoder, not reinforcement learning.
  3. Adversarial Multi-Task Model: A model employing an adversarial training scheme where the encoder is penalized for allowing the demographic attribute to be predicted from the learned features (minimizing mutual information), using a negative-squared Pearson correlation loss for age and binary cross-entropy for sex.
• Evaluation: Models were trained on the skewed datasets and evaluated on a balanced hold-out test set to isolate the effect of training bias. Performance was measured using Area Under the Curve (AUC) for classification and metrics such as Brier scores, Mean Absolute Error (MAE), and Pearson correlation for the auxiliary demographic prediction heads. Cross-dataset validation was performed without fine-tuning to assess generalizability.

Key Contributions

1. Extended Linear Programming Control: The study extends previous work by controlling both sex and age subgroups simultaneously, introducing five age groups and three skewed age distributions, whereas prior work focused primarily on sex.
2. Systematic Strategy Evaluation: The authors systematically compare single-task, reinforcing, and adversarial learning strategies across a granular range of demographic distributions, including extreme imbalances (e.g., 95% male/5% female).
3. Cross-Dataset Generalization: The study introduces a new external dermatoscopic dataset (DERM7PT) alongside the smartphone-based PAD-UFES-20, allowing for an analysis of how domain shifts (dermoscopic vs. smartphone) interact with demographic bias.
4. Auxiliary Head Analysis: The paper provides a detailed analysis of the auxiliary prediction heads in the reinforcing model, using their performance (AUC, Brier score, MAE, correlation) as a diagnostic tool to explain the success or failure of bias mitigation mechanisms.

Results

• Sex-Based Bias:
  • Data Imbalance Drives Bias: Sex-specific training datasets generally yielded the best performance for that specific sex. The baseline model exhibited significant bias, performing better on the majority sex in skewed datasets.
  • Mitigation Limitations: The reinforcing and adversarial models successfully narrowed or eliminated bias gaps in balanced and female-majority datasets. However, in male-majority settings, these strategies were less effective; models continued to perform better for males than females. The reinforcing model's auxiliary head failed to learn a robust discriminator in male-skewed scenarios (high Brier scores for the minority female group), leading to a collapse of the regularization effect.
  • Asymmetry: An asymmetry was observed where male-dominated training improved male performance significantly, but female-dominated training did not yield a comparable boost for female performance in the ISIC dataset, suggesting models may overfit to male-specific cues.
• Age-Based Bias:
  • Persistent Decline: Unlike sex bias, which was largely driven by data imbalance, age-related performance declines persisted even with balanced training. Younger age groups (A1: 0–50) consistently achieved the highest performance, while performance declined progressively in older age groups.
  • Shortcuts and Correlation: The auxiliary age-prediction head showed moderate correlation in balanced settings but strong correlation in older-skewed settings for the youngest cohort, suggesting the model relied on age-specific shortcuts when distributions were imbalanced.
  • Balanced Training Benefits: While balanced training improved performance for the A4 age category compared to younger-skewed training, it did not eliminate the overall trend of declining performance with age.
• Cross-Dataset Validation:
  • Domain Shift Impact: External validation on PAD-UFES-20 (smartphone) and DERM7PT (dermoscopic) revealed significant performance drops compared to internal validation, highlighting the impact of domain shift (imaging modality and acquisition protocol).
  • Bias Pattern Variation: The demographic bias patterns observed in the ISIC dataset did not fully replicate in external datasets. For instance, the adversarial model showed improved performance for female patients in female-dominated cohorts on the PAD-UFES-20 dataset, contrasting with ISIC results.

Significance and Claims
The paper claims that sex-related performance gaps in skin lesion classification are primarily driven by training set imbalances, which can be partially mitigated by multi-task and adversarial learning strategies, particularly when the minority class is sufficiently represented to train the auxiliary heads. In contrast, age-related performance disparities appear to stem from intrinsic biological or physiological differences (e.g., skin texture changes, wrinkles) that persist even with balanced sampling, suggesting that data rebalancing alone is insufficient for age fairness.

The authors emphasize that bias mechanisms are complex and dataset-dependent. They conclude that while their methods advance equitable AI by identifying specific sources of disparity, the interaction between explicit factors (data distribution, acquisition protocols) and implicit factors (biological variations, geographic differences) requires further disentangling. The study serves as a building block for developing more sophisticated healthcare AI systems by demonstrating that different mitigation strategies are required for different types of demographic bias.