Technical Summary: GESD – Beyond Outcome-Oriented Fairness

Problem Statement

Machine learning (ML) algorithms are increasingly deployed in high-stakes domains such as hiring, loan approvals, and criminal justice. While existing fairness metrics (e.g., statistical parity, equalized odds) effectively quantify disparities in model outcomes, they fail to address biases inherent in the decision-making procedure or the stability of model explanations. Current methods rely heavily on prediction statistics, potentially overlooking scenarios where a model produces equitable outcomes but relies on discriminatory or unstable internal reasoning. There is a critical gap in measuring whether members of different protected subgroups receive explanations of comparable quality, stability, and robustness.

Methodology

1. Group-level Explanation Stability Disparity (GESD)

The authors propose GESD, a procedural-oriented fairness metric designed to quantify disparities in the stability, robustness, and sensitivity of model explanations across protected subgroups. Unlike prior measures that compare average feature importance, GESD evaluates the reliability of explanations under input perturbations.

The calculation involves five steps:

• Explanation Aggregation: To ensure robustness against individual explainer inconsistencies, the method aggregates SHAP and LIME explanations into a single vector: $E_{agg}(f, x) = \frac{1}{2}(E_{SHAP} + E_{LIME})$.
• Perturbation: For each data point, perturbed versions are generated using Gaussian noise and feature masking (replacing features with baseline values). This simulates natural input fluctuations and missing data.
• Stability Measurement: The distance between the original explanation and the explanation of the perturbed input is calculated using the $L_1$ norm. A lower distance indicates higher stability. This is converted into a stability score $S(x)$ (e.g., $1/(1+\Delta(x))$).
• Group-wise Aggregation: Stability scores are averaged for each sensitive group (e.g., gender, race).
• Disparity Calculation:
  • For two groups ($K=2$), GESD is the absolute difference between group mean stability scores: $|S_0 - S_1|$.
  • For $K > 2$ groups, it is the variance of the group stability scores around the global mean.

2. FEU Optimization Framework

To address the multi-objective nature of balancing utility, outcome fairness, and explanation fairness, the authors introduce the Fairness–Explainability–Utility (FEU) framework.

• Objective Function: The problem is formulated as an unconstrained three-objective minimization:
  1. Fairness ($L_f$): Minimized Demographic Parity (DP) difference.
  2. Explainability ($L_e$): Minimized GESD.
  3. Utility ($L_u$): Maximized Area Under the Curve (AUC), converted to minimization via negation.
• Algorithm: The framework utilizes the Non-dominated Sorting Genetic Algorithm II (NSGA-II) to evolve a population of model configurations (hyperparameters like learning rate, epochs, and decision thresholds).
• Selection: From the resulting Pareto-optimal set, a final model is selected using Chebyshev scalarization to ensure balanced performance across all three objectives without extreme degradation in any single metric.

Key Contributions

1. Formulation of GESD: The paper defines a new procedural-oriented metric that quantifies group-level disparities in explanation robustness, stability, and sensitivity, independent of model outcomes.
2. FEU Framework: The authors propose an evolutionary multi-objective optimization framework that jointly optimizes for utility, outcome-based fairness, and explanation-based fairness.
3. Empirical Validation: The study demonstrates that GESD can detect hidden group-level explanation biases in both linear and non-linear models that outcome-based metrics miss.
4. Performance Trade-offs: The work shows that FEU achieves superior trade-offs between utility and fairness compared to state-of-the-art baselines (Adversarial Learning, Reduction, and Reweighing).

Experimental Results

Experiments were conducted on four benchmark datasets: German Credit, Recidivism, Math, and Portuguese. The proposed FEU method was compared against Adversarial Learning (ADL), Reduction (Red), and Reweighing (RW).

• Utility and Outcome Fairness: FEU generally outperformed or remained competitive with baselines in terms of AUC and F1 scores. It also achieved competitive results on outcome-oriented fairness metrics (DP and EOD), often outperforming baselines on the German Credit and Recidivism datasets.
• Procedural Fairness (GESD): While FEU showed slightly higher GESD values than linear baselines (Red, RW) on some datasets, this was attributed to the complexity of the neural network models used. Crucially, the results highlighted that models with high utility and outcome fairness can still exhibit significant explanation disparities.
• Model Sensitivity: The study found that non-linear models (like FEU and ADL) tend to generate less stable explanations across subgroups compared to linear models. Furthermore, the choice of explainer (SHAP vs. LIME) significantly impacted the detection of bias; aggregating both explainers (as done in GESD) was necessary to robustly capture disparities that single explainers might miss.
• Pareto Optimality: On the German Credit dataset, FEU generated non-dominated solutions, effectively balancing the three objectives, whereas linear baselines often sacrificed significant utility for marginal fairness gains.

Significance and Claims

The paper claims that GESD offers a comprehensive tool for diagnosing bias by bridging the gap between outcome-based and explanation-based fairness. Its primary significance lies in:

• Procedural Insight: It exposes procedural biases that traditional metrics overlook, ensuring that the reasoning behind decisions is as fair as the results.
• Model-Agnostic Utility: GESD is compatible with various learning algorithms and post-hoc interpretability methods.
• Holistic Optimization: The FEU framework demonstrates that it is possible to optimize for utility, fairness, and explainability simultaneously, providing a more holistic view of equitable modeling.

The authors note that while FEU performs well, it currently lacks specific bias-mitigation interventions (like adversarial debiasing) for extremely biased datasets, which is identified as a direction for future work. Additionally, they propose extending GESD to handle multiple protected categories (intersectionality) and shifting to many-objective optimization algorithms (e.g., NSGA-III) to manage the increased complexity.