Technical Summary: FairSAM: Fair Classification on Corrupted Image Data Through Sharpness-Aware Minimization

Problem Statement

Deep neural networks trained on clean data often suffer sharp performance degradation when deployed on corrupted test data (e.g., images affected by impulse noise, Gaussian noise, weather effects, or motion blur). Recent findings indicate that this degradation is not uniform; it disproportionately affects specific demographic subgroups, thereby amplifying algorithmic bias. While existing robust learning methods like Sharpness-Aware Minimization (SAM) improve overall generalization and resilience to noise, they fail to address the inequitable distribution of performance degradation across demographic groups. Conversely, standard fairness-aware methods often struggle to maintain robustness under data corruption. This creates an inherent tension between achieving robustness and ensuring fairness in corrupted environments.

Methodology: FairSAM

To address this tension, the authors propose FairSAM, a framework that integrates fairness-oriented strategies into the SAM optimization process. The methodology consists of three core components:

1. New Fairness Metric: Corrupted Degradation Disparity ($\Delta p$)

The authors formalize the problem by introducing a metric to quantify fairness specifically regarding performance degradation.

• Corrupted Degradation ($\Delta p_s$): Defined as the absolute difference in performance metrics between clean training data ($D_T$) and corrupted test data ($D_C$) for a specific demographic subgroup $s$: $\Delta p_s = |M(D_T^{s}, f) - M(D_C^{s}, f)|$.
• Corrupted Degradation Disparity ($\Delta p$): The absolute difference in degradation between the advantaged ($s^+$) and disadvantaged ($s^-$) groups: $\Delta p = |\Delta p_{s^+} - \Delta p_{s^-}|$.
  A lower $\Delta p$ indicates that the model's robustness is more equitably distributed across subgroups.

2. Instance-Reweighted SAM

Standard SAM minimizes the maximum loss within a neighborhood of parameters but treats all samples equally, often allowing majority groups to dominate the optimization. FairSAM introduces a reweighting mechanism to balance gradient contributions:

• Group Constraints: The method constrains the sum of weights for samples within each demographic group to a constant $c$, ensuring each group contributes equally to the objective regardless of its size.
• Sharpness-Based Reweighting: Within each group, samples with higher per-instance sharpness (larger maximum loss under perturbation) receive higher weights. This prioritizes "harder" examples within each group while maintaining group-level balance.

3. Efficient Per-Batch Perturbation

To avoid the computational cost of calculating individual adversarial perturbations for every sample (which would require $N$ forward passes per batch), FairSAM derives an efficient approximation:

• It approximates the weighted per-instance objective using a second-order Taylor expansion.
• It constructs a weighted batch loss $\ell_b(w) = \sum g_i \ell_i(w)$, where weights $g_i$ incorporate both group membership constraints and instance sharpness.
• A single per-batch perturbation $\epsilon^*$ is computed based on the gradient of this weighted batch loss, maintaining SAM's computational efficiency while enforcing fairness constraints.

Key Contributions

1. Problem Formulation: The paper identifies and formalizes the "robustness bias" challenge in image classification under data corruption, highlighting that standard robustness methods can inadvertently worsen fairness disparities.
2. Metric Proposal: Introduction of Corrupted Degradation Disparity to evaluate the fairness of performance degradation across demographic subgroups, distinct from one-shot fairness notions.
3. Framework Development: Proposal of FairSAM, the first framework to integrate fairness strategies into SAM. It utilizes instance reweighting and per-batch perturbation to simultaneously promote robustness and fairness.
4. Empirical Validation: Comprehensive experiments demonstrating that FairSAM achieves a superior trade-off between accuracy and fairness compared to baselines.

Experimental Results

The authors evaluated FairSAM on multiple real-world datasets (CelebA, FairFace, LFW, CheXpert) under various corruption types (snow, Gaussian noise, blur) and severity levels.

• Performance vs. Fairness Trade-off: Across imbalanced (CelebA) and balanced (FairFace) datasets, FairSAM consistently achieved the lowest Corrupted Degradation Disparity ($\Delta p$) among all baselines, including Vanilla SAM, GroupSAM, FairReg, and GroupDRO.
• Accuracy Preservation: Unlike some fairness methods that sacrifice overall accuracy, FairSAM maintained competitive or superior corrupted accuracy. For instance, on CelebA with Gaussian noise, FairSAM achieved the highest corrupted accuracy (0.9291) while minimizing $\Delta p$ (0.0073).
• Robustness to Asymmetry: In experiments where privileged groups faced less severe noise than disadvantaged groups (asymmetric noise), FairSAM maintained the lowest degradation disparity across different backbones (ResNet-18 and DINOv3).
• Out-of-Distribution (OOD) Generalization: When models trained on one dataset (e.g., CelebA) were tested on another (e.g., LFW), FairSAM demonstrated the lowest $\Delta p$, indicating better equitable generalization compared to GroupSAM, which risked shifting the advantaged group into a disadvantaged position.
• Hyperparameter Sensitivity: The framework was found to be robust to hyperparameter choices, with $\rho=0.05$ (perturbation radius) and $c=1$ (group weight budget) providing the best balance.

Significance and Claims

The paper claims that FairSAM provides a structured solution for fair and robust image classification in the presence of data corruption. Its primary significance lies in addressing a critical blind spot in robustness research: the assumption that aggregate robustness improvements benefit all groups equally.

The authors argue that:

• Robustness and fairness are interconnected; robust models often exhibit disparate performance degradation across demographic groups, which can perpetuate societal inequalities if unaddressed.
• Fairness-aware optimization during training can achieve equitable outcomes without significant sacrifices in overall accuracy.
• The benefits of this approach extend beyond imbalanced datasets to balanced ones, suggesting that performance disparities under corruption are driven by algorithmic biases rather than just data distribution issues.

The work positions itself as a step toward safer AI deployment in high-stakes applications (e.g., healthcare, criminal justice) where systems must maintain fairness under real-world, noisy conditions. The authors acknowledge limitations, noting the current focus on binary sensitive attributes in image classification, and suggest future work should explore intersectional groups and other domains.