Technical Summary: Bias Mitigation in Face Recognition via Demographic-based Supervised Contrastive Learning

1. Problem Statement

Face recognition (FR) systems, particularly those based on deep learning, exhibit significant bias across demographic groups (e.g., gender, age, ethnicity). This bias manifests as disparate error rates, where certain groups suffer higher False Match Rates (FMR) or False Non-Match Rates (FNMR). While training data imbalance is a known contributor, simply balancing the training dataset does not fully mitigate the issue.

The core challenge lies in the operational deployment of FR systems, which typically function at very low false match rates (the "tail" of the non-match score distribution). Standard class-balancing techniques often improve the mean of score distributions but fail to address the behavior in the tail, where fairness is most critical. Furthermore, existing mitigation strategies often face a trade-off: methods that remove sensitive demographic information from embeddings to prevent bias often degrade overall verification performance, while post-hoc score normalization requires demographic information during deployment, which is not always available or desirable.

2. Methodology: DeSCon

The authors propose Demographic-based Supervised Contrastive loss (DeSCon), a framework that integrates supervised contrastive learning into standard margin-based softmax frameworks (e.g., ArcFace). The approach aims to improve fairness by explicitly targeting the tail of the non-match score distribution without sacrificing discriminative power.

Core Mechanism

DeSCon combines two complementary objectives:

1. Centroid-based Loss (e.g., ArcFace): Pulls embeddings toward their class center and pushes them away from other class centers to ensure inter-class separability.
2. Supervised Contrastive Loss (SupCon): Operates directly on embeddings, pulling "mated" pairs (same identity) together and pushing "non-mated" pairs (different identities) apart.

The key innovation is the demographic-aware pair selection within the SupCon component. Unlike standard contrastive learning which treats all non-mated pairs equally, DeSCon guides the selection of negative pairs based on demographic labels to specifically target within-group separation.

Sampling Strategies

The paper explores three specific strategies for selecting non-mated pairs ($N(a)$) for the contrastive loss denominator:

• DeSCon-All: Applies contrastive loss to all non-mated pairs in the batch, regardless of demographic group.
• DeSCon-WG (Within-Group): Restricts non-mated pair sampling to samples belonging to the same demographic group as the anchor. This enhances local balance and intra-group separation.
• DeSCon-Hard: Focuses on the hardest non-mated pairs within the same demographic group (the top $K$ most similar non-mated samples). This strategy specifically targets the tail of the non-match distribution, aiming to maximize fairness regularization where it is most needed.

Training Objective

The total loss function combines ArcFace and the demographic-weighted SupCon:
$$L_{total} = \frac{1}{B} \sum_{b=1}^{B} w_d [L_{arc}(x_b) + \lambda \cdot L_{sup}(x_b)]$$
Where $w_d$ is a group-aware weight to compensate for demographic imbalance in the training data, and $\lambda$ balances the two objectives. The framework is designed to be compatible with various backbones and margin-based losses (e.g., ArcFace, AdaFace).

3. Key Contributions

• Introduction of DeSCon: A fairness-aware loss function that augments centroid-based losses with supervised contrastive learning to jointly enhance accuracy and fairness.
• Strategic Pair Sampling: The exploration of three sampling strategies (All, WG, Hard) to balance global separation, within-group consistency, and hard negative regularization.
• Comprehensive Evaluation: Extensive experiments across multiple backbones (IResNet50, IResNet100) and benchmarks (RFW, BFW, VGGFace2, IJB-C) using both balanced and imbalanced training regimes.
• Standardized Metrics: A rigorous evaluation adhering to ISO/IEC 19795-10 standards, utilizing the Gini coefficient to measure inequality in FMR (FPD) and FNMR (FND) across groups, rather than relying solely on per-group accuracy with separate thresholds.

4. Experimental Results

The authors evaluated DeSCon against state-of-the-art bias mitigation methods (e.g., RamFace, MixFairFace, DeFT, ScoreNorm) and baselines.

• Fairness vs. Performance: DeSCon variants consistently improved fairness metrics (reduced FPD and FND) while maintaining competitive verification performance (TMR).
• Superiority of DeSCon-Hard: Across balanced and imbalanced training scenarios, DeSCon-Hard emerged as the most robust variant. It achieved the best balance between verification performance and fairness, delivering consistent reductions in both FPD and FND.
  • On the RFW dataset (balanced test), DeSCon-Hard reduced FPD significantly compared to the ArcFace baseline.
  • On VGGFace2 (imbalanced test), DeSCon-Hard maintained robust fairness improvements where other methods showed mixed results.
• Ablation Findings:
  • Replacing SupCon with other pairwise losses (e.g., Triplet, Multi-Similarity) degraded performance, confirming the necessity of the SoftMax-based aggregation over all non-mated pairs for shaping the distribution tail.
  • Hard negative mining (DeSCon-Hard) was identified as the most effective sampling strategy.
  • The method generalized well to stronger margin-based losses like AdaFace.
• Benchmark Performance: On standard verification benchmarks (LFW, CPLFW, AgeDB, IJB-C), DeSCon variants maintained accuracy comparable to the ArcFace baseline, with some improvements observed on the challenging IJB-C dataset.

5. Significance and Claims

The paper claims that DeSCon offers a practical solution to the fairness-performance trade-off in face recognition. Its significance lies in:

1. Tailored Fairness: By focusing on the "hard" non-mated pairs within demographic groups, the method directly addresses the operational tail of the score distribution, which is critical for real-world deployment.
2. No Inference Overhead: Unlike post-hoc normalization methods, DeSCon does not require demographic information or additional computations during the inference (verification) phase.
3. Robustness: The approach remains effective even when training data is demographically imbalanced, a common real-world scenario.
4. Methodological Rigor: The authors emphasize the importance of adhering to international standards (ISO/IEC 19795-10) for fairness evaluation, arguing that many existing metrics (like per-group accuracy with separate thresholds) provide overly optimistic and unrealistic assessments of fairness.

The authors conclude that while DeSCon-Hard provides the most robust fairness improvements, the choice of sampling strategy and base loss are critical design decisions for fairness-aware training. They note that future work should focus on extending this approach to large-scale datasets (e.g., WebFace-260M) where demographic annotations are currently unavailable or unreliable.