💻 computer science

Reliable Skin-Tone Fairness in Dermatology AI: From Lesion-Level Leakage to Tone-Adaptive Learning

This paper addresses skin-tone fairness in dermatology AI by exposing the severe performance inflation caused by lesion-level data leakage and the unreliability of luminance-based skin-tone proxies, ultimately proposing a rigorous evaluation protocol and a tone-adaptive preprocessing pipeline that significantly improves model sensitivity for darker skin tones under realistic clinical conditions.

Original authors: Muhammad Rizwan Tahir, Shaista Habib, Sadaf Jabeen, Zahoor Khan

Published 2026-10-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Muhammad Rizwan Tahir, Shaista Habib, Sadaf Jabeen, Zahoor Khan

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Skin cancer is a formidable threat, but when caught early, it is often treatable. The key to survival lies in spotting the warning signs quickly, a task that has increasingly fallen to artificial intelligence. Computers trained on thousands of medical images can now identify suspicious skin spots with a level of accuracy that rivals human experts. However, a troubling pattern has emerged in these digital doctors: they often work well for people with lighter skin but struggle significantly with those who have darker skin. This gap is not just a technical glitch; it is a matter of life and death, as missed diagnoses on darker skin can lead to later detection and worse outcomes. The core of the problem involves two hidden traps. First, the data used to train these systems often contains multiple photos of the exact same skin spot, which can trick the computer into memorizing the specific spot rather than learning to recognize the disease. Second, the way researchers have traditionally measured skin tone in these studies has often been unreliable, relying on how bright or dark an image looks rather than the actual color of the person's skin.

A team of researchers set out to untangle these issues and build a more fair system for diagnosing skin lesions across all skin tones. They began by exposing a fundamental flaw in how many studies are conducted. In the world of skin cancer datasets, it is common to have several pictures of a single mole or lesion taken from slightly different angles. When researchers split this data into a training set and a test set without realizing that the same lesion appears in both, the computer effectively uses information from the test set during training. It sees the lesion during its training and then gets tested on a slightly different view of that same lesion, leading to artificially high scores that do not reflect real-world performance. The team demonstrated that this "data leakage" can inflate performance metrics by a significant margin, making a flawed system look successful. More dangerously, this practice destabilizes the measurements used to check for fairness, meaning a system could appear to treat all skin tones equally simply because the test was rigged. To fix this, they developed a new, strict method for splitting data that ensures no single lesion ever appears in both the training and testing phases, creating a truly honest test of the computer's abilities.

With a fair testing ground established, the researchers introduced a new approach to help the computer see better on darker skin. They recognized that lesions on darker skin often have lower contrast, making them harder to distinguish from the surrounding tissue, especially in photos taken with ordinary cameras under variable lighting. To address this, they created a pipeline that selectively brightens and sharpens the images of darker skin tones before the computer analyzes them, while leaving lighter skin images unchanged. They also added a strategy to ensure the computer sees enough examples of rare and dangerous types of skin spots during its training. When they tested this system on standardized medical images taken with special dermatoscope cameras, the results were surprisingly modest; the extra processing did not help much, and in some cases, it slightly reduced the ability to spot the most dangerous cancers. This suggested that for high-quality, controlled medical images, the extra processing was unnecessary.

However, the story changed when they tested the system on real-world clinical photos taken with smartphones or standard cameras, where lighting is often uneven and unpredictable. In these conditions, the system that selectively enhanced the darker skin images showed a clear improvement. The ability to correctly identify dangerous lesions on darker skin rose noticeably, closing a gap that had previously existed. The researchers found that this improvement came primarily from the contrast enhancement, which made the hidden details of the lesions visible, rather than just from seeing more examples of them. They also discovered that a common shortcut used in the past—guessing a person's skin tone based on how bright their photo looks—was completely unreliable. In one test, this brightness measure failed to match the actual skin type assigned by doctors, proving that such shortcuts can lead to incorrect conclusions about fairness.

The study concludes that building truly fair artificial intelligence for skin cancer requires more than just better algorithms; it demands better science. The researchers showed that without strict controls to prevent data leakage and without accurate ways to measure skin tone, claims of fairness are often illusions. Their work suggests that the most effective way to help patients with darker skin is not a one-size-fits-all solution, but a targeted approach that adapts to the specific conditions of the image. By enhancing the visibility of lesions in variable lighting, the system can offer a genuine boost in accuracy where it is needed most. The team has released their tools and methods to the public, hoping that other scientists will adopt these stricter standards. In doing so, they aim to ensure that the next generation of medical AI does not just look good on paper, but actually works for every patient, regardless of their skin tone.

Technical Summary: Reliable Skin-Tone Fairness in Dermatology AI

Problem Statement
Dermatology AI systems frequently exhibit reduced sensitivity for patients with darker skin tones (Fitzpatrick IV–VI), a disparity driven by limited representation in training data and reduced lesion visibility in darker skin. While deep learning models have achieved dermatologist-level accuracy on average, these benefits are not equitably distributed. The paper identifies two critical methodological flaws in current fairness evaluations that obscure or distort these disparities:

Lesion-Level Data Leakage: Standard image-level train/test splits often place multiple images of the same lesion in both sets. This allows models to memorize specific lesions rather than learning generalizable features, artificially inflating performance metrics and destabilizing subgroup recall estimates.
Invalid Tone Proxies: In the absence of clinician-assigned skin tone labels, researchers often use image luminance (e.g., median CIELAB $L^*$ of the image border) as a proxy. The authors demonstrate that this proxy fails to correlate with actual skin tone, measuring lighting and exposure conditions instead.

Methodology
The study introduces a tone-adaptive pipeline and a rigorous leak-free evaluation protocol to assess fairness interventions across three datasets: HAM10000 (dermoscopic), PAD-UFES-20 (clinical smartphone), and Fitzpatrick17k (clinical).

Evaluation Protocol: The authors replace standard image-level splits with a lesion-grouped, class-stratified split (using StratifiedGroupKFold). This ensures that all images of a single lesion (or near-duplicate group) reside in only one partition (train, validation, or test), eliminating leakage. They quantify leakage rates in standard splits, finding them as high as 39% (HAM10000).
Tone-Adaptive Pipeline: The proposed intervention consists of two components applied selectively to the "Dark" subgroup (defined by $L^*$ $L^{*}$ proxy for HAM10000 and clinician labels for others):
1. Selective Contrast Enhancement: Applying Contrast-Limited Adaptive Histogram Equalization (CLAHE) specifically to the $L^*$ channel of darker-skin images to improve lesion visibility.
2. Risk-Aware Augmentation: A novel oversampling strategy that balances class imbalance and subgroup representation. The augmentation factor is weighted by clinical risk ( $R_y$ ) and visibility difficulty ( $W_y$ ), targeting high-risk, low-visibility classes (e.g., melanoma) while leaving common, visible classes (e.g., nevi) unchanged.
Experimental Design: The pipeline is evaluated using a $2 \times 2$ ablation study (Baseline, CLAHE-only, Augmentation-only, Full) across two architectures (ResNet-50 and EfficientNet-B0). The study also compares the pipeline against standard fairness interventions (class reweighting, focal loss, subgroup reweighting, Group-DRO).

Key Contributions

Leak-Free Evaluation Framework: The authors establish that conventional image-level splits inflate macro-F1 scores by up to 0.10 and destabilize subgroup recall. They release leakaudit, an open-source tool to detect lesion-level leakage and generate group-disjoint splits.
Proxy Validity Analysis: The study empirically demonstrates that luminance-based proxies (border $L^*$ ) do not track clinician-assigned Fitzpatrick types (Spearman $\rho = -0.003$ , $p=0.91$ ), rendering luminance-stratified analyses unreliable for skin-tone fairness claims.
Modality-Dependent Interventions: The research isolates the specific conditions under which tone-adaptive preprocessing is effective.
Rigorous Statistical Reporting: The study adopts bootstrap confidence intervals, McNemar tests, and equivalence tests (TOST) to ensure reported gains are statistically robust.

Results

Impact of Leakage: On HAM10000, a standard image-level split inflated baseline macro-F1 from 0.672 to 0.774. Controlled leakage experiments showed a monotonic increase in accuracy (6.6 points) as duplicate overlap increased from 0% to 40%, while simultaneously increasing noise in darker-skin recall estimates.
Performance on Standardized Dermoscopy (HAM10000): Under leak-free evaluation, the full pipeline provided no clear benefit and showed a slight, non-significant decline in darker-skin recall (0.772 to 0.732). The pipeline did not improve calibration and increased image noise. Augmentation alone performed better than the full pipeline, suggesting contrast enhancement is unnecessary for controlled dermoscopic lighting.
Performance on Clinical Images (PAD-UFES-20): The pipeline improved darker-skin recall from 0.655 to 0.714 (+5.9 percentage points). Ablation analysis revealed that CLAHE was the primary driver of this gain (raising recall to 0.702), while augmentation provided complementary support. This suggests contrast enhancement is most beneficial where lighting is variable and lesion contrast is low.
Performance on Fitzpatrick17k: Darker-skin recall was already high (0.890), leaving little room for improvement; the pipeline maintained this balance without significant changes.
Malignant Lesion Detection: The approach showed encouraging gains in malignant-lesion recall for darker skin on PAD-UFES-20 (0.611 to 0.713) and HAM10000 (0.500 to 0.550), though the latter was a marginal increase.

Significance and Claims
The paper argues that reliable assessment of skin-tone fairness in dermatology AI is critically dependent on valid evaluation protocols and accurate tone measurement.

Methodological Necessity: Claims of fairness improvements are often artifacts of data leakage or invalid subgroup definitions. A model evaluated with leaky splits can appear beneficial even if it offers no real generalization.
Contextual Efficacy: Tone-adaptive preprocessing (specifically contrast enhancement) is not a universal fix. It is most effective in variable-lighting clinical settings (e.g., primary care, teledermatology) where lesion visibility is compromised, but offers limited value in standardized dermoscopy.
Trade-offs: The study openly reports that contrast enhancement increases image noise and does not improve model calibration, suggesting these are areas for future optimization.

The authors conclude that future work must prioritize lesion-grouped splits, validated skin-tone labels, and modality-specific interventions to ensure equitable diagnostic support.

Technical Summary: Reliable Skin-Tone Fairness in Dermatology AI

More like this