🤖 AI

Beyond Predictive Fairness: Quantifying Attribution Consistency Across Demographic Groups in Diabetic Retinopathy Screening

This paper introduces the Explanation Consistency Score (ECS) to quantify the similarity of attribution maps across demographic groups in diabetic retinopathy screening, revealing that while predictive performance varies by ethnicity, the visual evidence used by models remains consistent, thereby demonstrating that predictive fairness and explanation consistency are complementary dimensions of model behavior.

Original authors: Kerol Djoumessi, Philipp Berens

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kerol Djoumessi, Philipp Berens

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of medical imaging, computers have become remarkably skilled at spotting diseases in photographs of the human body. They can scan images of the retina, the light-sensitive layer at the back of the eye, and identify signs of diabetic retinopathy, a condition that can lead to blindness if left untreated. For these systems to be trusted in a hospital, they must be fair. This means they should work just as well for a patient from one background as they do for a patient from another. Traditionally, doctors and researchers have checked for this fairness by looking at the final score: did the computer get the diagnosis right? If it missed the disease in one group of people but caught it in another, that is a problem. However, getting the right answer is only half the story. A computer could arrive at the correct diagnosis for the wrong reasons, perhaps by noticing a subtle artifact in the image that happens to be more common in a specific group, rather than by actually seeing the disease itself. To truly trust these tools, we need to know not just what they decide, but how they see the world.

A team of researchers at the University of Tübingen in Germany set out to investigate this deeper layer of understanding. They asked a simple but profound question: when a computer looks at an eye image to find diabetic retinopathy, does it focus on the same parts of the eye for everyone, regardless of their ethnicity? To find the answer, they turned to a massive collection of retinal images from the EyePACS dataset, which included photographs from patients of five major ethnic backgrounds: Latin American, African descent, Indian origin, Caucasian, and Asian. They trained a standard computer vision system to distinguish between healthy eyes and those showing early signs of the disease. Once the system was ready, the researchers did not just check if it was right or wrong; they asked the system to show its work. Using a technique that highlights the specific areas of an image that influenced the decision, they created visual maps showing exactly where the computer was looking.

The researchers then developed a new way to measure how similar these visual maps were across different groups. They compared the average attention patterns of the computer for each ethnicity to see if the focus shifted depending on who the patient was. They found a striking disconnect between performance and attention. The computer's ability to correctly identify the disease varied significantly across the groups. For instance, it was much better at spotting the disease in patients of Indian origin, with an AUC of 0.83, compared to an AUC of 0.54 for Caucasian patients. This is a substantial gap in predictive fairness. Yet, when the researchers looked at the visual maps, the story changed completely. The computer was looking at the same spots on the retina for every group. Whether the patient was of Indian, African, or Caucasian descent, the system consistently focused on the same retinal features to make its decision. The similarity in these visual patterns was extremely high, with consistency scores ranging from 0.85 to 0.92 on a scale where one represents perfect agreement.

To ensure that these results were not simply because some groups had more severe cases of the disease than others, the researchers repeated the analysis by grouping patients strictly by the severity of their condition. Even when they compared patients with the exact same stage of disease, the computer continued to look at the same regions for everyone. The consistency remained high, suggesting that the difference in performance was not caused by the computer using different logic or looking at different clues for different people. Instead, the computer seemed to be applying the same visual reasoning to all patients, even though it was less successful at applying that reasoning to certain groups.

This discovery suggests that the two main ways we usually judge fairness in artificial intelligence are actually measuring different things. A system can be fair in how it thinks, focusing on the same medical evidence for everyone, while still being unfair in its results, failing to diagnose some groups as often as others. The researchers found no strong link between how well the computer performed and how consistently it looked at the images. This means that simply making a model more accurate for one group might not automatically fix the way it sees the world, and conversely, a model that looks at the right things might still struggle to get the right answers for everyone. The study concludes that to build truly trustworthy medical AI, we must look beyond just the final score. We need to understand the reasoning behind the decision, ensuring that the computer is not just guessing correctly, but is seeing the disease in the same way for every single patient.

Technical Summary: Beyond Predictive Fairness

Problem Statement
In medical imaging, particularly diabetic retinopathy (DR) screening, fairness is predominantly evaluated through predictive performance metrics (e.g., sensitivity, specificity, AUC) across demographic subgroups. While these metrics quantify disparities in outcomes, they fail to address whether models rely on consistent visual evidence across different demographic groups. Recent studies indicate that models can encode demographic information even when trained for clinical tasks, and two models with similar subgroup performance may utilize fundamentally different visual reasoning. This gap raises a critical question: do predictive disparities necessarily imply differences in model reasoning (attribution) across demographic groups? Existing literature lacks quantitative analyses of attribution differences across demographics and their relationship with predictive fairness.

Methodology
The authors introduce a quantitative framework to investigate the demographic consistency of visual attribution patterns using retinal fundus images from the EyePACS dataset.

Dataset and Subgroups: The study utilized 90,595 high-quality images from the EyePACS dataset, stratified by patient ethnicity (Latin American, African Descent, Indian Origin, Caucasian, Asian) and DR severity grades (0–4). The analysis focused on a binary classification task: distinguishing healthy eyes (Grade 0) from eyes with any stage of DR (Grades 1–4).
Model: A ResNet-50 architecture, initialized with ImageNet weights, was trained to perform the binary classification.
Attribution Generation: Visual evidence underlying predictions was extracted using two distinct attribution methods: SmoothGradCAM++ (gradient-based) and Score-CAM (gradient-free). Attribution maps were normalized to form spatial probability distributions.
Explanation Consistency Score (ECS): The core contribution is the ECS, a metric based on the Jensen–Shannon (JS) divergence to quantify the similarity of attribution distributions between demographic groups.
- Global ECS: Computed as $1 - JS(\mu_{g1} \parallel \mu_{g2})$ , where $\mu_g$ is the mean attribution distribution for group $g$ .
- Conditional ECS: To control for confounding effects of disease severity, the metric was computed conditionally on specific DR grades (specifically Grades 1 and 2), comparing attribution patterns only among patients with identical disease stages.
Experimental Design: The study analyzed correctly classified samples to isolate consistency under successful decision-making. Performance was evaluated via AUC, sensitivity, and specificity, while consistency was measured via ECS and within-group consistency (WG-ECS).

Key Results

Predictive Disparities: Significant performance gaps were observed across ethnicities. Patients of Indian origin achieved the highest AUC (0.92) and sensitivity (0.83), while Caucasian patients exhibited the lowest AUC (0.83) and sensitivity (0.54), resulting in a 23% sensitivity gap. Notably, performance did not correlate linearly with subgroup size or disease prevalence.
High Attribution Consistency: Despite predictive disparities, explanation consistency remained uniformly high across all demographic groups. ECS values ranged from 0.89 to 0.92 for Score-CAM and 0.85 to 0.88 for SmoothGradCAM++.
Decoupling of Metrics: The study found no significant statistical association between predictive performance and explanation consistency. Groups with lower predictive performance (e.g., Caucasian) maintained attribution patterns highly similar to other groups.
Conditional Analysis: When controlling for disease severity (analyzing only Grade 1 and Grade 2 cases), high ECS values persisted. Attribution patterns were found to be more stable for Grade 2 (more advanced lesions) than Grade 1, suggesting that visual clarity of lesions influences attribution stability, but demographic consistency remains robust regardless of severity.
Variability: Within-group attribution variability was comparable across demographics, confirming that the high global ECS values were not artifacts of averaging but reflected genuine consistency.

Key Contributions

Introduction of ECS: The proposal of the Explanation Consistency Score, a fairness-aware metric based on Jensen–Shannon divergence, to quantitatively measure the similarity of attribution maps across demographic subgroups.
Conditional Formulation: A method to compute ECS conditionally on disease severity to disentangle attribution consistency from confounding factors related to disease prevalence and stage distribution.
Empirical Evidence of Decoupling: The demonstration that predictive fairness and explanation consistency capture complementary, rather than equivalent, dimensions of model behavior. The findings suggest that models can exhibit significant performance disparities across ethnicities while relying on largely consistent visual evidence.

Significance and Claims
The paper argues that relying solely on predictive metrics provides an incomplete picture of fairness in medical AI. The findings motivate fairness evaluations that extend beyond predictive performance to include the consistency of model reasoning. The authors claim that predictive disparities do not necessarily imply differences in model attribution behavior; thus, a model may be "unfair" in its outcomes but "consistent" in its visual reasoning, or vice versa. This distinction is crucial for trustworthy deployment, as it highlights that improving predictive fairness may require addressing factors beyond the visual evidence the model attends to, such as data distribution shifts or disease manifestation differences across populations. The study acknowledges limitations, including the focus on correctly classified samples, the group-level nature of the metric, and the use of post-hoc attribution methods, suggesting future work should explore individual-level variability and self-explainable models.

Technical Summary: Beyond Predictive Fairness

More like this