Technical Summary: Toward Calibrated, Fair, and Accurate Deepfake Detection

Problem Statement

Deepfake detectors currently exhibit significant performance disparities across demographic groups. While aggregate accuracy may appear high, specific subgroups (e.g., females, African males, or darker skin tones) suffer from substantially higher false-positive rates (FPR) or lower true-positive rates (TPR). These gaps pose severe deployment risks: content moderation systems may disproportionately flag authentic videos from minority groups as fake, and identity verification systems may fail more frequently for specific demographics. Furthermore, adversaries can exploit these blind spots to target vulnerable populations with detection-evading deepfakes.

Existing mitigation strategies face critical limitations for practical deployment:

• Pre-processing: Data balancing or synthetic generation often discards valuable samples, requires significant manual effort, and suffers from domain gaps when synthetic data is used.
• In-processing: Methods that modify training objectives (e.g., fairness losses, reweighting) typically require full model retraining and explicit demographic labels, which are often unavailable in production environments.
• Post-processing: Existing threshold adjustment methods often require group labels at inference time or sacrifice overall accuracy to achieve fairness. Additionally, methods tuned for specific datasets often fail to generalize to unseen generators.

There is a lack of solutions that are demographic-label-free, retraining-free, and capable of improving both fairness and overall accuracy simultaneously.

Methodology: The Face-Fairness (FF) Framework

The authors propose Face-Fairness (FF), a plug-and-play framework comprising three variants designed to mitigate bias without retraining the base detector or requiring demographic annotations.

1. Face-Feature Tuning (FFT)

FFT is the primary contribution and the first demographic-label-free fairness method for deepfake detection.

• Core Insight: Pre-trained face embeddings (e.g., from ArcFace) implicitly encode visual patterns correlated with detector failures. These embeddings serve as proxies for protected attributes without requiring explicit labels.
• Mechanism: FFT employs a lightweight, two-layer Multi-Layer Perceptron (MLP) that acts as a conditional calibrator. It takes the concatenation of the frozen base detector's logit output ($Z(x)$) and the frozen face embedding ($\phi(x)$) as input.
• Function: The MLP learns a non-linear mapping to remap the logits ($\tilde{Z}(x)$), effectively learning a residual correction $r_\psi(\phi(x), Z(x))$. This allows the model to adjust decision boundaries conditionally based on the embedding space, correcting systematic biases in specific "neighborhoods" of the feature space while preserving reliable decisions elsewhere.
• Training: The base detector and face encoder remain frozen. Only the lightweight MLP is trained on a held-out validation set using binary cross-entropy.

2. FF-Max (Group-Conditional Threshold Optimization)

This variant is used when demographic labels are available during calibration and inference.

• Mechanism: It learns group-specific decision thresholds ($t_g$) to maximize the worst-group accuracy.
• Optimization: It solves a max-min optimization problem: $\max_{t} \min_{g} \text{Acc}_g$. Thresholds are enumerated on a grid and fixed for test time based on known group labels.

3. FF-Discover (Clustering-Based Fair Thresholding)

This variant operates without any demographic labels.

• Mechanism: It discovers latent groups by applying K-means clustering to the L2-normalized face embeddings.
• Optimization: Similar to FF-Max, it learns cluster-specific thresholds to maximize the minimum accuracy across the discovered clusters. The number of clusters ($K$) is selected via silhouette analysis.

Key Contributions

1. FFT (Face-Feature Tuning): A novel, label-free, retraining-free post-processing method that improves both worst-group accuracy and overall accuracy by learning a conditional calibrator on top of frozen face embeddings.
2. FF-Max & FF-Discover: Two practical variants that optimize worst-group accuracy using either explicit labels (FF-Max) or embedding-derived latent clusters (FF-Discover), serving as strong baselines for deployment scenarios where neural network training is a barrier.
3. Detector Agnosticism: The framework is compatible with any deepfake detector backbone (demonstrated on Xception and MobileNetV3-Small) and adds negligible runtime overhead.
4. Complementarity: The framework can be stacked with existing in-processing fairness methods to correct residual biases that survive fairness-aware training.

Experimental Results

The authors evaluated the framework on OpenForensics (in-domain) and FaceForensics++ (FF++) (cross-dataset, leave-one-manipulation-out) using Xception and MobileNetV3-Small backbones.

• In-Domain Performance (OpenForensics):

  • FFT consistently reduced the False Positive Rate (FPR) and True Positive Rate (TPR) gaps across 14 gender/ethnicity groups.
  • It improved the Worst-Group Accuracy (WGA) while simultaneously increasing overall accuracy (Acc@0.5) and AUC.
  • On MobileNetV3-Small, FFT achieved the best AUC (0.9919) and WGA (0.9395) among post-processing methods, with no "leveling down" (i.e., no group suffered accuracy degradation compared to the baseline).
  • FFT outperformed other post-processing baselines (Fair-FLIP, BPFA) and matched or exceeded in-processing methods (DAG-FDD, DAW-FDD) in fairness metrics without requiring retraining.

• Cross-Dataset Performance (FF++ Leave-One-Out):

  • In the challenging F2F hold-out setting, where detectors often collapse to random guessing, FFT significantly boosted accuracy from near-random (0.4140) to 0.6392 on Xception.
  • While some alternative methods showed reduced fairness gaps, the authors argue this was due to predictions collapsing toward non-discriminative scores (regression to chance). FFT was the only post-processing method to achieve above-chance accuracy gains while improving utility.
  • FF-Discover achieved the highest WGA (0.4800) among all methods in this setting, demonstrating that embedding-derived clusters capture fairness-critical structure even without labels.

• Calibration:

  • Calibration curves showed FFT substantially reduced Expected Calibration Error (ECE) compared to the baseline and other post-processing methods, which often degraded calibration.

Significance and Claims

The paper claims that Face-Feature Tuning (FFT) offers a novel solution to the deployment challenge of deepfake detection by achieving fairness without demographic labels, retraining, or accuracy sacrifice.

• Practicality: The method is highly practical for organizations using commercial base models, as it requires no access to model internals, no demographic annotations, and minimal computational overhead.
• Privacy: By avoiding the need for explicit demographic labels, FFT is particularly valuable in privacy-sensitive contexts where collecting such data is legally or ethically problematic.
• Mechanism: The effectiveness stems from the ability of face embeddings to provide rich, continuous demographic information that captures subtle variations beyond discrete categories, allowing the calibrator to learn non-linear corrections for complex interactions between appearance and detector behavior.
• Limitations: The authors modestly note that the method assumes face embeddings sufficiently encode demographic information, which may not hold for heavily occluded or low-quality images, and requires a representative calibration set.

In conclusion, the work demonstrates that leveraging the structure of pre-trained face embeddings allows for the correction of systematic biases in deepfake detectors, enabling more equitable and accurate deployment in real-world scenarios.