Technical Summary: Adaptive Calibration for Fair and Performant Facial Recognition

Problem Statement

Modern facial recognition systems map face images to high-dimensional embeddings and utilize cosine similarity as a matching score. While these systems have achieved high accuracy, they exhibit significant demographic disparities, with error rates often an order of magnitude higher for individuals with dark skin tones compared to those with lighter skin.

The core issue identified is a fundamental mismatch between the training objective and system operationalization. Training typically employs distance-based objectives (e.g., contrastive or angular-margin losses) that ensure correct matches are closer than incorrect ones but make no assumptions about absolute distances. Consequently, the same cosine distance can correspond to different probabilities of a correct match depending on the region of the embedding space. Underrepresented demographic groups often occupy lower-density regions of this space where embedding behavior is less constrained. When a global threshold is applied to raw cosine distances, it fails to account for these local variations, leading to disproportionate errors (false rejections or false acceptances) for minority groups. Existing post-processing methods often rely on global monotonic mappings (like Platt scaling) or discrete clustering heuristics that fail to capture continuous, region-specific calibration needs, or they require demographic metadata at inference, which is often unavailable in real-world deployments.

Methodology: Adaptive Calibration (AC)

The authors propose Adaptive Calibration (AC), a post-hoc calibration strategy that maps cosine similarity scores and local embedding context to well-calibrated probabilities without requiring demographic group annotations at training or inference.

Core Concept

AC addresses the mismatch by learning a continuous, locally varying calibration function. It posits that the probability of a match depends not only on the cosine similarity ($s$) but also on the location of the pair within the embedding space.

Pair Representation

To capture local context, AC utilizes the normalized average embedding ($\bar{z}$) of a face pair:
$$\bar{z} = \frac{z_1 + z_2}{\|z_1 + z_2\|_2}$$
where $z_1$ and $z_2$ are the normalized embeddings of the two faces. This representation is chosen because:

1. It is symmetric (order of probe/gallery does not matter).
2. It resides on the same unit hypersphere as the backbone embeddings, preserving geometric properties.
3. It varies continuously, avoiding the discrete jumps associated with cluster-based methods.

Model Variants

AC trains a secondary classifier to predict the match probability $p$ given the pair representation and similarity score. Three variants are proposed:

1. AC-MLP: A lightweight Multi-Layer Perceptron mapping the concatenated vector $[\bar{z}, s]$ to a probability via a sigmoid output.
2. AC-Linear: A logistic regression model on $[\bar{z}, s]$. This preserves the local ranking of the similarity score while allowing for region-specific shifts.
3. AC-Density: A hybrid approach that incorporates a local density measure. It calculates the mean Euclidean distance to the $k$-nearest neighbors in the training set ($\rho(\bar{z})$) to identify sparse regions. It then applies a ridge regression to predict the residual error of a base Platt-calibrated score, combining the base score, local density, and the linear features.

Training and Inference

The method is trained on a calibration split using binary cross-entropy to minimize the difference between predicted probabilities and ground-truth match labels. Crucially, it requires no demographic labels during training or inference. Theoretically, if the groups are recoverable from the embedding space, a sufficiently expressive model trained to minimize cross-entropy will be group-calibrated for any such group.

Key Contributions

1. Novel Calibration Strategy: Introduction of Adaptive Calibration, which maps local embedding context and similarity scores to calibrated match probabilities, addressing shortcomings in both accuracy and fairness.
2. Superior Performance: Demonstration that AC dominates existing post-hoc calibration techniques (such as FairCal, FALCON, and FRAPP´E) across a wide range of datasets (RFW, BFW, LFW) and backbones (ArcFace, AdaFace, MagFace, FaceNet).
3. Label-Free Fairness: The method achieves equitable calibration without requiring demographic group annotations at inference, making it practical for deployment where such metadata is unavailable.
4. Avoidance of "Leveling Down": Unlike some fairness interventions that degrade performance for majority groups to equalize outcomes, AC improves overall calibration and accuracy, particularly for the groups where the system performs worst.
5. Generalizability: The approach provides group calibration for any group recoverable from the embedding space (e.g., ethnicity, gender, or other attributes like headwear), not just pre-defined demographic axes.

Results

The paper evaluates AC against state-of-the-art baselines on three standard benchmarks:

• Datasets: Racial Faces in-the-Wild (RFW), Bias in Face Verification (BFW), and Labeled Faces in the Wild (LFW).
• Backbones: ArcFace (R50, R100), AdaFace, MagFace, and FaceNet.
• Metrics:
  • Accuracy: Global True Positive Rate (TPR) at a False Positive Rate (FPR) of $10^{-3}$.
  • Fairness: Worst-group Area Under the ROC Curve (AUROC) and Worst-group Brier Score.
  • Leveling-Down: A metric (LU-AUROC) measuring how many of the worst-performing groups improve without degrading others.

Key Findings:

• Accuracy and Fairness: AC-Linear and AC-Density consistently achieve the best or near-best label-free results. For instance, on RFW with MagFace, AC-Density improved the worst-group AUROC from 94.19 to 96.02 (a ~30% reduction in residual error) without degrading other groups.
• Calibration Quality: AC significantly reduces the Brier score for the worst-calibrated groups compared to baselines like FairCal and FSN.
• Leveling-Down: AC variants frequently achieve the maximum "Leveling-Up" score (where all groups improve), whereas methods like FRAPP´E and FSN often fail to improve the worst groups without harming others.
• Comparison to Demographic-Aware Methods: While methods requiring true demographic labels (Oracle, GST, DemoNorm) serve as upper bounds, AC approaches their performance in many settings without needing those labels at inference.

Significance and Claims

The paper claims that Adaptive Calibration provides a practical solution for equitable facial recognition. By converting distances into calibrated probabilities using a locally varying function, AC corrects the fundamental mismatch between training objectives and operational thresholds.

The authors emphasize that their approach:

• Improves Overall Performance: It does not merely redistribute errors but often enhances global accuracy.
• Avoids "Leveling Down": It improves performance for the worst-served groups without sacrificing the performance of better-served groups, addressing a common failure mode in fairness interventions.
• Is Deployable: It requires no demographic metadata at inference, making it suitable for real-world applications where such data is not available.
• Is Flexible: It can be integrated into existing verification pipelines with minimal custom code, particularly the linear variants.

The work suggests that by focusing on local embedding context, systems can achieve robust fairness and accuracy simultaneously, offering a path toward more equitable deployment of facial recognition technology.