🧬 biology

FairLogue: Evaluating Intersectional Fairness across Clinical Machine Learning Use Cases using the All of Us Research Program

This paper introduces FairLogue, a toolkit that applies intersectional fairness auditing to clinical machine learning models using the All of Us dataset, revealing that while combined demographic analyses uncover larger disparities than single-axis approaches, counterfactual diagnostics suggest most observed biases are consistent with random group membership rather than inherent model discrimination.

Original authors: Nick Souligne, Vignesh Subbian

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nick Souligne, Vignesh Subbian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The Big Picture: Why We Need a New Lens

Imagine you are a doctor trying to predict who might get sick. You build a computer program (an AI) to help you. But, you realize that sometimes the AI makes mistakes more often for certain groups of people.

For a long time, researchers checked for these mistakes by looking at one trait at a time. They would ask, "Is the AI unfair to Black people?" or "Is the AI unfair to women?" separately.

The Problem: This is like checking if a car is safe by only looking at the tires, then only looking at the brakes, but never checking how they work together. In real life, people have many identities at once (e.g., a Black woman, or an elderly Hispanic man). When you look at these traits separately, you miss the unique, "double-whammy" disadvantages that happen when they combine.

The Solution: The authors created a toolkit called FairLogue. Think of it as a high-powered microscope that lets you zoom in on the specific combinations of identities (intersectionality) to see hidden biases that the "single-lens" view misses.

The Experiment: Two Real-World Tests

To test this microscope, the researchers used a massive, diverse database called "All of Us" (which is like a giant library of health records from millions of Americans). They tested the toolkit on two different medical scenarios:

The Bleeding Test: Predicting if a patient taking common antidepressants (SSRIs) would have a dangerous bleeding event.
The Stroke Test: Predicting if a patient with an irregular heartbeat (Atrial Fibrillation) would have a stroke within two years.

What They Found: The "Shadow" Effect

When they looked at the data with the new FairLogue microscope, they found something surprising:

The Single-Lens View was Blurry: When they looked at race or gender alone, the AI seemed mostly fair. The gaps in performance were small.
The Intersectional View was Sharp: When they looked at the combinations (e.g., Black women vs. White men), the gaps got much bigger. It was like realizing that while the car's tires are fine and the brakes are fine, the car handles terribly when it's raining and the road is icy. The combination of factors created a bigger problem than either factor alone.

The Analogy: Imagine a school grading system.

If you check if the system is fair to "Boys," it looks okay.
If you check if it's fair to "Girls," it looks okay.
But if you check "Girls who are also new to the country," you might find they are failing because the system doesn't account for their specific struggle. The "FairLogue" toolkit found these hidden struggles in the medical data.

The Twist: Is the AI Actually "Racist" or "Sexist"?

Here is the most fascinating part. Just because the AI makes more mistakes for a specific group, does that mean the AI is intentionally biased against them?

To answer this, the researchers used a Time-Travel Simulation (Counterfactual Analysis).

The Analogy: Imagine you have a bag of marbles. Some are red, some are blue. You want to know if the bag is rigged against the red marbles.
The Test: They took the data and "shuffled the deck." They kept all the medical facts (age, health, meds) exactly the same, but they randomly swapped the race and gender labels.
The Result: They asked, "If we randomly assigned these people to different groups, would the AI still make the same mistakes?"

The Surprise: In both medical tests, the answer was mostly "Yes."
Even when they randomized the groups, the AI still made similar mistakes. This suggests that the AI isn't necessarily "hating" a specific group. Instead, the AI is picking up on other clues in the data (like specific health conditions, medication history, or socioeconomic factors) that happen to be more common in those groups.

The Metaphor: It's like a weather app that predicts rain.

If the app predicts rain for everyone in a valley, but it's actually sunny, the app isn't "hating" the valley. It's just reacting to the low pressure system that happens to be over the valley.
Similarly, the AI wasn't discriminating based on race; it was reacting to complex health patterns that were unevenly distributed across different groups.

The Takeaway: Why This Matters

Don't Just Look at One Thing: If you only check for bias based on race or gender alone, you are missing the real story. You need to look at how these identities mix.
Bias is Often a Symptom, Not the Disease: The study shows that the AI's unfairness often comes from the data (the real-world inequalities in healthcare) rather than the AI being "evil." The AI is just a mirror reflecting the messy reality of the world.
The Toolkit is a Detective: FairLogue helps doctors and engineers figure out why the AI is struggling. Is it because the AI is broken? Or is it because the real world is unfair?
- If the AI is broken, we fix the code.
- If the real world is unfair (e.g., certain groups have less access to care), we need to fix the healthcare system, not just the computer.

In a Nutshell

This paper says: "We built a better magnifying glass (FairLogue) to find hidden unfairness in medical AI. We found that looking at race and gender separately hides the real problems. However, when we simulated a 'what if' world where race and gender were random, we realized the AI isn't necessarily the villain—it's often just reacting to deep, complex inequalities in our healthcare system that we need to fix."

1. Problem Statement

Healthcare machine learning models often perpetuate systemic inequities because they are trained on data reflecting historical biases. Traditional fairness evaluations typically assess demographic attributes (e.g., race or gender) in isolation (single-axis analysis). This approach fails to capture intersectional bias, where individuals belonging to multiple underrepresented groups (e.g., Black women) experience compounded disparities that are obscured when groups are analyzed separately.

Furthermore, existing tools often lack the ability to distinguish whether observed disparities are caused by the demographic group membership itself (direct discrimination) or by complex correlations between demographics and other covariates (e.g., socioeconomic status, comorbidities) embedded in the data. There is a critical need for a toolkit that operationalizes intersectional fairness auditing and utilizes counterfactual analysis to diagnose the root causes of bias in clinical settings.

2. Methodology

The study utilized the All of Us Research Program (Registered Tier V8) dataset, chosen for its focus on underrepresented populations and large scale, which supports robust intersectional subgroup analysis.

Experimental Design:
The authors replicated and evaluated two distinct clinical prediction tasks using the FairLogue toolkit:

Case Study A: Prediction of Selective Serotonin Reuptake Inhibitor (SSRI)-associated bleeding events (within 90 days of exposure).
- Models: LightGBM and Random Forest.
- Cohort: 12,850 participants stratified by four SSRIs (citalopram, escitalopram, fluoxetine, sertraline).
Case Study B: 2-year stroke risk prediction in patients with Atrial Fibrillation (AF).
- Model: LightGBM.
- Cohort: 11,160 participants. Notably, this model included prior bias mitigation techniques (disparity-aware tuning and post-processing calibration) to test FairLogue's ability to detect residual bias.

Analytical Framework:

Observational Metrics: The study computed standard fairness metrics across single-axis (race, gender) and intersectional (race $\times$ $\times$ gender) subgroups:
- Demographic Parity (DP)
- Equalized Odds (EO) - specifically False Positive Rate (FPR) gaps.
- Equal Opportunity Difference (EOD) - True Positive Rate (TPR) gaps.
- Unfairness Values ("u-values"): Calculated under an acceptable threshold of 0.10 (10%).
Counterfactual Analysis: To determine if disparities were attributable to group membership, the authors performed a counterfactual simulation where intersectional group membership was randomized.
- They compared observed disparities against the "counterfactual scenario" (randomized group assignment).
- If observed disparities significantly exceeded those in the counterfactual scenario, it suggested direct discrimination by the model. If they were comparable, it suggested disparities arose from underlying data distributions (covariates) rather than the model encoding bias.

3. Key Contributions

Validation of FairLogue: The study provides broad validation of the FairLogue toolkit across multiple distinct clinical use cases, demonstrating its utility beyond a single application.
Intersectional vs. Single-Axis Analysis: It empirically demonstrates that intersectional analyses consistently reveal larger and more complex disparities than single-axis analyses, which often underestimate model inequities.
Diagnostic Distinction: The paper introduces a methodological distinction between observational disparities (what is seen in the data) and attribution of bias (whether the model discriminates based on identity). By using counterfactual analysis, the authors show that many observed disparities are comparable to those expected under randomized group membership, implying the bias stems from data structure rather than explicit model discrimination.
Metric Sensitivity: The study highlights that in low-prevalence clinical tasks, False Positive Rate metrics may be less informative due to conservative model boundaries, whereas True Positive Rate (Equal Opportunity) disparities provide a more critical signal of subgroup performance differences.

4. Key Results

Case Study A (SSRI Bleeding Events):

Observational: Intersectional analysis revealed significantly larger disparities than single-axis analysis. While Demographic Parity and FPR gaps were minimal, Equal Opportunity Difference (EOD) showed substantial variability across subgroups.
Counterfactual: The analysis revealed minimal unfairness attributable to group membership.
- Positive prediction unfairness (u-values) was effectively zero.
- Negative prediction unfairness showed modest levels, but confidence intervals for counterfactual error rates (cFNR and cFPR) overlapped substantially across subgroups.
- Conclusion: The models treated intersectional subgroups similarly when predicting positives; observed disparities were likely driven by correlations between identity and other covariates (e.g., comorbidities) rather than direct demographic discrimination.

Case Study B (Stroke Risk in AF):

Observational: Despite prior bias mitigation techniques (tuning for subgroup AUROC differences), intersectional analysis still revealed larger disparities than gender-only analysis.
Counterfactual: The empirical cumulative distribution functions (ECDFs) of disparity differences were concentrated near zero and well below the 0.10 threshold.
Conclusion: The magnitude of unfairness attributable specifically to intersectional group membership was minimal. The observed disparities were consistent with what would be expected under randomized group assignment, suggesting the bias is structural (data-driven) rather than algorithmic.

General Findings:

Model Architecture Independence: Fairness patterns were consistent across LightGBM and Random Forest models, suggesting the bias is driven by data distribution rather than specific model architectures.
Low Prevalence Impact: In tasks with low positive outcome prevalence (like SSRI bleeding), models adopt conservative decision boundaries, minimizing FPR disparities but potentially masking TPR (True Positive Rate) inequities.

5. Significance and Implications

Beyond "Bias Detection": The study argues that simply identifying disparities is insufficient. Practitioners must use counterfactual diagnostics to determine why disparities exist. If disparities persist under randomized group membership, mitigation strategies should focus on data curation, feature engineering, or addressing structural inequities (e.g., comorbidity prevalence) rather than just adjusting model thresholds.
Intersectionality is Mandatory: Single-axis fairness audits are inadequate for clinical ML. Intersectional auditing is necessary to uncover hidden inequities affecting vulnerable subgroups.
Tooling for Practitioners: FairLogue provides a structured workflow for clinicians and data scientists to move from "Is the model fair?" to "Is the model fair, and if not, is the fault the model or the data?"
Future Directions: The authors suggest integrating FairLogue directly into model development and deployment pipelines. Future work should investigate the specific covariate relationships driving these disparities to enable targeted bias mitigation strategies.

In summary, the paper demonstrates that while clinical ML models often exhibit intersectional disparities, these are frequently artifacts of the underlying healthcare data structure rather than explicit algorithmic discrimination. The FairLogue toolkit is essential for making this distinction, thereby guiding more effective and ethical interventions in healthcare AI.