Worst-Group Equalized Odds Regularization for Multi-Attribute Fair Medical Image Classification

Technical Summary: Worst-Group Equalized Odds Regularization for Multi-Attribute Fair Medical Image Classification

1. Problem Statement

The paper addresses the critical issue of systematic performance disparities in medical AI across demographic groups (e.g., age, sex, race). While aggregate metrics like Area Under the Curve (AUC) often appear similar across groups, they can mask clinically significant inequities at fixed inference-time operating points. Specifically, some subgroups may exhibit over-diagnostic behavior (elevated True Positive Rates [TPR] and False Positive Rates [FPR]), while others show under-diagnostic behavior (reduced TPR and FPR). These opposing tendencies can cancel out in aggregate AUCs, obscuring meaningful disparities in clinical decision-making.

Existing mitigation strategies face several limitations:

• Objective Misalignment: Methods like GDRO, JTT, or FairBatch often reweight samples or modify log-loss to reduce average predictive risk, which may fail to capture localized subgroup disparities at specific operating points.
• Single-Attribute Focus: Most prior work addresses fairness for a single demographic attribute, leaving intersectional disparities unaddressed or exacerbated.
• Intersectional Challenges: Approaches handling multiple attributes often require coarsened subgroup definitions to maintain statistical power or struggle with instability in representation-level debiasing (e.g., suppressing mutual information) for small subgroups.
• Setting Limitations: Many methods are evaluated in simplified binary classification settings, limiting their applicability to the multi-label scenarios common in clinical imaging.

2. Methodology

The authors propose a Worst-Group Equalized Odds (EO) Margin Regularizer designed for realistic multi-label diagnostic settings. The method targets operating-point level disparities using threshold-free logit margins.

Core Formulation:

• Problem Setup: The dataset consists of inputs $x_i$, labels $y_i$, and demographic attributes $a_i$ (e.g., age, sex, race). Subgroups $g$ are defined by attribute-value pairs.
• Worst-Group Identification: In each mini-batch, the method dynamically identifies the worst-performing subgroups without maintaining separate constraints per attribute:
  • For positives ($y=1$), it identifies the subgroup $g_{min}^{[+]}$ with the lowest mean predicted probability (indicating under-diagnosis risk).
  • For negatives ($y=0$), it identifies the subgroup $g_{max}^{[-]}$ with the highest mean predicted probability (indicating over-diagnosis risk).
• Logit Margin Definition: The goal is to enforce separation between the worst-case positive samples of the under-diagnosed group and all negative samples, and vice versa for the over-diagnosed group.
  • Ideally: $\min_{m: y_m=1, g_m=g_{min}^{[+]}} \ell_m > \max_{n: y_n=0} \ell_n$ and $\max_{m: y_m=0, g_m=g_{max}^{[-]}} \ell_m < \min_{n: y_n=1} \ell_n$.
• Differentiable Relaxation: Since hard min/max operators are non-differentiable, the authors replace them with Log-Sum-Exp (LSE) operators. This smooths the extrema, allowing samples close to the worst-case to contribute to the penalty, which aids in mitigating intersectional violations.
  • $margin_{EO+} = \text{LSE}_{max}(\{\ell_i : y_i=0\}) - \text{LSE}_{min}(\{\ell_i : y_i=1, g_i=g_{min}^{[+]}\})$
  • $margin_{EO-} = \text{LSE}_{max}(\{\ell_i : y_i=0, g_i=g_{max}^{[-]}\}) - \text{LSE}_{min}(\{\ell_i : y_i=1\})$
• Loss Function: A hinge loss is applied to penalize only positive margins (violations):
  • $L_{EO+} = \max(0, margin_{EO+})$
  • $L_{EO-} = \max(0, margin_{EO-})$
  • The final objective is $L = L_{base} + \lambda_{EO+}L_{EO+} + \lambda_{EO-}L_{EO-}$, where $L_{base}$ is the standard binary cross-entropy (or averaged across classes for multi-label).

Key Mechanism: The regularizer acts as a localized AUC optimization, strengthening the ranking of weak positives from the worst-performing subgroup against all negatives, and conversely for negatives, without requiring post-hoc threshold adjustments.

3. Key Contributions

1. Unified Multi-Attribute Regularizer: A method that identifies the "worst" subgroup dynamically across all demographic axes (age, sex, race) simultaneously, avoiding the need for explicit intersectional constraints or coarsened subgroup definitions.
2. Bidirectional Penalty: Unlike standard log-loss which primarily improves AUC via re-ranking, this bi-directional regularizer explicitly penalizes both over-diagnosis (FPR side) and under-diagnosis (TPR side) at the operating point.
3. Differentiable Worst-Group Optimization: The use of LSE-based margins enables stable gradient-based training for worst-group fairness, distributing gradient flow across samples rather than concentrating on single outliers.
4. Multi-Label Applicability: The approach is validated in realistic multi-label diagnostic settings, addressing a gap in existing literature which often focuses on binary classification.

4. Experimental Results

The method was evaluated on two medical imaging datasets:

• RNFL-OCT: An ophthalmic dataset (3,300 images) for glaucoma classification.
• MIMIC-CXR: A large chest radiograph dataset (82,282 images) with a multi-label task for Pleural Effusion, Cardiomegaly, and Atelectasis.

Performance:

• Fairness Metrics: The proposed method consistently achieved the lowest EOdds (worst-to-best group ratio for TPR/FPR disparity) and highest EOM (Equality of Opportunity across Multiple Subclasses) across both datasets and all demographic attributes (Age, Race, Sex).
  • On RNFL-OCT, it reduced joint EOdds to 0.226 (vs. 0.354 for baseline) and increased joint EOM to 0.926.
  • On MIMIC-CXR, it reduced joint EOdds to 0.127 (vs. 0.254 for baseline) and increased joint EOM to 0.916.
• Accuracy Trade-off: The method achieved these fairness improvements with minimal impact on overall AUC.
  • RNFL-OCT: $\Delta AUC = -0.006$.
  • MIMIC-CXR: $\Delta AUC = -0.007$.
• Comparison: Compared to baselines like FairBatch, GDRO, JTT, Adversarial Training, and FaMI, the proposed method offered a superior fairness-accuracy balance. Other methods either yielded weaker parity improvements or incurred significantly larger accuracy losses (e.g., FaMI showed $\Delta AUC \approx -0.06$).

Ablation: Experiments confirmed that both the positive ($L_{EO+}$) and negative ($L_{EO-}$) regularization components are complementary; enabling both yields the strongest fairness gains.

5. Significance and Claims

The paper claims that its approach provides a unified mechanism to address fairness across multiple demographic axes without the computational or statistical overhead of managing separate constraints for each attribute or intersectional group. By targeting localized score discrepancies responsible for over- and under-diagnostic behaviors, the method mitigates worst-case subgroup disparities while preserving diagnostic performance.

The authors emphasize that while AUC can mask clinically relevant disparities at the operating point, their regularizer effectively optimizes the ranking in the specific score regions that drive TPR and FPR disparities. This results in a more favorable fairness-accuracy trade-off compared to existing methods, making it particularly suitable for clinical imaging scenarios where multi-label predictions and strict demographic parity are critical.