Technical Summary: DuetFair and FairDRO for Fair Medical Image Segmentation

1. Problem Statement

Medical image segmentation models often exhibit uneven performance across different patient subgroups (e.g., defined by race, age, or tumor stage). While existing fairness methods aim to improve average performance across these subgroups, they typically treat each subgroup as an internally homogeneous unit. This approach creates a blind spot the authors term "intra-subgroup hidden failure."

In this scenario, a subgroup may appear to perform well on average because difficult, high-loss cases within that subgroup are masked by easier cases. Consequently, the model fails to optimize for the most challenging patients within a specific demographic or clinical group, leading to unreliable patient-level predictions even when group-level metrics seem acceptable. The paper argues that fairness in medical segmentation requires addressing two distinct but coupled axes:

1. Inter-subgroup heterogeneity: Adapting to distributional differences between subgroups.
2. Intra-subgroup variation: Ensuring robustness against difficult cases within each subgroup.

2. Methodology: DuetFair and FairDRO

To address these dual challenges, the authors propose the DuetFair mechanism, a dual-axis framework, and instantiate it as FairDRO (Fair Distributionally Robust Optimization).

The DuetFair Mechanism

DuetFair conceptualizes fairness as a joint problem requiring:

• Inter-subgroup adaptation: The model must account for distributional shifts across different groups.
• Intra-subgroup robustness: The model must maintain reliability for difficult cases within each group, preventing them from being "washed out" by subgroup averages.

The FairDRO Implementation

FairDRO couples two specific components to realize the DuetFair mechanism:

1. Inter-Subgroup Axis: Distribution-Aware Mixture-of-Experts (dMoE)

   • Goal: Address inter-subgroup heterogeneity at the representation level.
   • Mechanism: Instead of relying solely on loss weighting, FairDRO uses a shared backbone with a subgroup-conditioned mixture of experts. Given an input image $x_i$ and subgroup attribute $g_i$, intermediate features are adapted via a weighted sum of expert transformations selected by a routing network.
   • Effect: This allows the model to learn subgroup-specific feature representations, reducing the mismatch between the model and diverse subgroups before the loss is even computed.

2. Intra-Subgroup Axis: Subgroup-Conditioned Distributionally Robust Optimization (DRO)

   • Goal: Address intra-subgroup variation and hidden failures.
   • Mechanism: Rather than minimizing the empirical average loss within a subgroup ($R_g$), FairDRO minimizes the worst-case risk within a subgroup-specific ambiguity set.
   • Formulation: For each subgroup $g$, the robust risk $R_g^{rob}$ is defined as the supremum of expected loss over a set of distributions $Q_g$ close to the empirical distribution $\hat{P}_g$ (measured by KL-divergence).
   • Effect: This optimization naturally up-weights high-loss samples within each subgroup, ensuring that difficult cases are not ignored even if the subgroup is small or heterogeneous.

The Objective Function:
The final objective aggregates these subgroup-specific robust risks:
$$\min_{\theta, \phi} L_{FairDRO} = \sum_{g \in \mathcal{G}} w_g R_g^{rob}(\theta, \phi)$$
where $w_g$ is a normalized weight (set to $1/|\mathcal{G}|$ in the implementation), ensuring uniform aggregation of subgroup risks without manual up-weighting of specific groups.

3. Key Contributions

• Identification of Intra-Subgroup Hidden Failure: The paper identifies that averaging subgroup performance masks severe patient-level errors, motivating a dual-axis approach to fairness.
• DuetFair Mechanism: A conceptual framework that explicitly separates and couples inter-subgroup adaptation and intra-subgroup robustness.
• FairDRO Algorithm: A concrete implementation combining dMoE for representation learning and subgroup-conditioned DRO for loss aggregation. This design allows the model to adapt across groups while prioritizing hard samples within each group.
• Comprehensive Evaluation: Evaluation across three benchmarks covering demographic, clinical, and institutional attributes.

4. Experimental Results

The authors evaluated FairDRO on three datasets: Harvard-FairSeg (fundus images, race-based), HAM10000 (skin lesions, age-based), and an in-house 3D Radiotherapy Target Dataset (prostate CT, T-stage and institution-based).

• Harvard-FairSeg (Race): FairDRO achieved the best equity-scaled performance (ES-Dice and ES-IoU). It significantly improved the Dice and IoU for the weakest subgroup (Asian) compared to strong baselines like dMoE and Prompt-GDRO, while maintaining comparable average performance.
• HAM10000 (Age): FairDRO recovered performance for the minority group (Age $\ge$ 80), improving Dice and IoU over dMoE. However, the authors note that since the age-based gap was primarily inter-group, the marginal gain from intra-group robustness was smaller compared to other settings.
• 3D Radiotherapy Dataset:
  • Tumor Stage (T1-T4): FairDRO showed the most dramatic improvements here. It significantly improved the worst-group Dice for rare stages (T1 and T4) and reduced the spread of per-patient Dice scores within each stage. Qualitative analysis showed FairDRO corrected systematic over-extension or under-delineation errors seen in baselines.
  • Institution: In a realistic deployment scenario involving data shift across hospitals, FairDRO was the only method to substantively recover the worst-performing site (Gangnam Severance Hospital), reducing the performance gap between the best and worst institutions while preserving performance on the source site.

5. Significance and Claims

The paper claims that FairDRO offers a more holistic approach to fairness by acknowledging that subgroup averages can be misleading. By coupling representation learning (to handle group differences) with robust optimization (to handle within-group difficulty), FairDRO ensures that:

1. Models do not sacrifice minority group performance for average accuracy.
2. Difficult cases within any subgroup (majority or minority) are not obscured by the group mean.

The authors state that FairDRO achieves its strongest gains in settings with substantial patient-level heterogeneity within subgroups (e.g., the radiotherapy dataset), where standard group-level methods fail to address the "hidden failures." They modestly acknowledge that when subgroup labels already explain most of the performance disparity (as seen in the age-based HAM10000 results), the additional value of intra-subgroup robustness is less pronounced, but the method remains competitive.

Ultimately, the work positions DuetFair and FairDRO as steps toward clinical AI systems that provide equitable care, ensuring that the benefits of medical AI extend to all patients, including those with difficult or rare presentations within any demographic or clinical group.