FairSelect: A Systematic Evaluation of Multi-Level and Intersectional Algorithmic Fairness

Technical Summary of FairSelect: A Systematic Evaluation of Multi-Level and Intersectional Algorithmic Fairness

Problem Statement
The field of algorithmic fairness has produced numerous methods to identify and mitigate bias in machine learning models across pre-processing, in-processing, and post-processing stages. However, current practices suffer from three primary limitations:

1. Isolation: Fairness techniques are typically evaluated in isolation along single demographic axes, rather than in combination or across intersectional subgroups.
2. Lack of Systematic Guidance: Model developers lack clear guidance on how to select or combine mitigation strategies, particularly when addressing multiple sources of bias simultaneously.
3. Context Dependency: Interventions developed in one context often rely on assumptions (causal relationships, data structures) that do not generalize to new domains, such as healthcare, where intersectional disparities can have direct implications for patient outcomes.

Existing tools (e.g., AIF360, Fairlearn) generally support individual mitigation strategies rather than the structured, concurrent evaluation of multiple techniques across the modeling lifecycle. Furthermore, assessments limited to single attributes often fail to detect inequities affecting intersectional populations.

Methodology
To address these gaps, the authors present FairSelect, a modular Python-based toolkit designed to systematically evaluate combinations of fairness techniques. The framework operates through a structured execution sequence involving three primary phases:

1. Pre-processing: Modifying training data distributions.
2. In-processing: Incorporating fairness constraints during model optimization.
3. Post-processing: Adjusting model outputs or thresholds after training.

Key Components:

• Mitigation Techniques: The toolkit implements 12 specific techniques (3 pre-processing, 5 in-processing, 4 post-processing), including Outcome-Group Reweighting, SMOTE, Exponentiated Gradient Reduction, and Group-Specific Youden Thresholding.
• Model Support: It supports seven supervised learning algorithms (Logistic Regression, Random Forest, Decision Tree, Neural Network, SVM, XGBoost, LightGBM).
• Evaluation Metrics: The system evaluates predictive performance (AUROC, Accuracy, F1, Brier Score, ECE) and fairness (Demographic Parity, Equalized Odds, TPR, FPR) across both individual demographic attributes and intersectional subgroup combinations.

Experimental Design
The framework was validated using two distinct approaches:

1. Synthetic Data Experiments: A series of datasets were generated to simulate specific bias mechanisms (e.g., representation imbalance, label bias, calibration bias) and data imbalances. These served as controlled stress tests to verify if specific interventions corrected their targeted bias patterns.
2. Real-World Clinical Replication: The toolkit was applied to a real-world dataset predicting 2-year stroke risk among patients with atrial fibrillation (AF) using data from the All of Us Research Program. The study utilized a development cohort (8,777 participants) and a testing cohort (2,383 participants) with distinct geographic and temporal distributions to evaluate subgroup generalizability.

Key Results

• Synthetic Experiments: Targeted fairness methods generally reduced intended subgroup disparities. Combined multi-level strategies produced larger average fairness improvements (mean EO_diff improvement of 0.0331) compared to single methods (0.0175), though this came with modest utility tradeoffs (mean AUROC decrease of 0.0125 for multi-level vs. 0.0014 for single). Crucially, results showed that fairness interventions interact in non-additive ways; some combinations yielded limited benefits or reduced fairness gains relative to single methods.
• Clinical Stroke Prediction:
  • Baseline: Models showed strong predictive performance but substantial variability in fairness (e.g., Decision Tree baseline EO_diff of 1.000 vs. Random Forest at 0.063).
  • Single-Method Performance: Fairness improvements were inconsistent. Only 22.3% of single-method runs reduced Equalized Odds difference (EO_diff), and many actually worsened disparities.
  • Multi-Level Performance: Combined strategies showed higher potential for improvement but also high variability. While many combinations were ineffective or counterproductive, specific configurations (e.g., Random Forest with Reweighting and Ensemble In-processing) achieved meaningful fairness gains while maintaining predictive performance. Conversely, some combinations significantly reduced AUROC.
  • Intersectionality: The study highlighted that disparities often persist or emerge at the intersections of multiple protected characteristics, which single-axis evaluations might miss.

Significance and Claims
The paper argues that fairness interventions are not independent components and cannot be assumed to have cumulative or additive effects. The primary significance of FairSelect lies in its ability to:

1. Systematically Evaluate Combinations: It provides a practical framework for testing how fairness strategies interact across the modeling lifecycle, revealing that effectiveness is highly context-dependent.
2. Support Intersectional Analysis: It enables the evaluation of disparities across intersecting identities, which is critical for clinical decision-making where heterogeneous patient populations exist.
3. Reframe Fairness-Utility Tradeoffs: The findings suggest that fairness and predictive performance are not always strictly competing objectives. In some cases, multi-level interventions corrected bias patterns that also degraded model generalization, thereby improving both equity and accuracy.

The authors conclude that there is no "one-size-fits-all" fairness solution. Instead, systematic, multi-level evaluation frameworks like FairSelect are necessary to identify effective mitigation strategies that improve subgroup equity while preserving model performance in real-world clinical settings.