CounterFace: A Synthetic Face Dataset for Fine-Grained Counterfactual Evaluation of Face Recognition Systems

Technical Summary: CounterFace

Problem Statement
Face Recognition (FR) systems are widely deployed but often exhibit biases and sensitivity to specific factors such as lighting, pose, and skin tone. Standard evaluation benchmarks (e.g., LFW, CelebA) typically rely on real-world datasets that capture coarse-grained variations (aging, pose) but lack fine-grained, controlled variations in facial attributes like hairstyles, makeup, or accessories. While counterfactual evaluation offers a method to isolate the impact of specific attributes on model robustness, existing synthetic counterfactual datasets suffer from significant limitations. Previous approaches either rely on noisy, unverified synthetic data or require costly human verification to select compliant samples. This human-in-the-loop process limits the scale of attribute and demographic coverage, preventing a comprehensive analysis of FR failure modes across diverse populations.

Methodology
The authors propose CounterFace, a fully automated pipeline for generating a synthetic counterfactual face dataset. The core of the methodology is a Rejection Sampling (RS) framework that integrates off-the-shelf image generators with custom automated verifiers.

1. Generation Pipeline (Stage 1):

   • Source Faces: The pipeline generates source faces using diffusion models (Realistic Vision) and GANs. To ensure identity consistency across variations, the authors use GPT-4o to compile celebrity names for specific demographics, generating multiple variations per identity.
   • Attribute Modification: Source faces are modified to add or remove specific attributes using editing techniques. The authors employ SEGA (diffusion-based) and StyleCLIP (GAN-based), selecting the optimal method for each of the 20 target attributes based on preliminary testing.
   • Candidate Pairs: This stage produces a large pool of candidate face pairs (source vs. modified) for each attribute-demographic combination.

2. Rejection Sampling (Stage 2):

   • Candidates are filtered against three strict counterfactual requirements:
     • Validity: The modified face must be semantically correct, retaining the original identity and free of unnatural artifacts.
     • Correctness: The target attribute must be successfully added or removed.
     • Specificity: Only the target attribute should change; all other attributes must remain constant.
   • Automated Verifiers:
     • Artifact Detectors: A Linear SVM trained on CLIP embeddings (for diffusion/SEGA outputs) and a Vision-Language Model (MOLMO-72B for GAN/StyleCLIP outputs) detect unnatural distortions.
     • Attribute Detectors: Proprietary Vision-Language Models (GPT-4o, Claude 3.5 Sonnet, Gemini-Pro) are used to verify the presence/absence of attributes. The best-performing model is selected for each attribute-demographic combination based on a human-annotated validation set.
   • Identity Preservation: Crucially, the pipeline does not use an FR model to verify identity preservation, avoiding bias toward specific recognition systems. Instead, identity consistency is ensured through rigorous hyperparameter tuning of the editing methods.

Key Contributions

• CounterFace Dataset: A new synthetic dataset comprising 11,821 counterfactual face pairs. It covers 20 facial attributes (e.g., facemask, heavy makeup, facial hair, sunglasses) and 8 demographic factors (combinations of 2 sexes and 4 ethnicities), totaling 160 attribute-demographic combinations. This represents an expansion of 14 attributes and 2 demographics over prior synthetic datasets.
• Automated Verification Pipeline: A novel, fully automated rejection sampling pipeline that eliminates the need for human verification, significantly reducing costs and enabling scalability.
• Faithfulness Validation: A post-hoc user study ($n=900$) confirms the dataset's quality, showing that 96.9% of sampled pairs retained identity and 84.04% met all counterfactual requirements.
• Comprehensive FR Evaluation: The dataset is used to evaluate six FR systems (two commercial: AWS Rekognition, Face++; four open-source: AdaFace, MagFace, ArcFace, FaceNet) across all 160 combinations.

Results

• System Performance Variance: Performance degradation varies significantly across attributes and demographics for all six systems. Newer models (AdaFace, MagFace) and commercial systems (AWS, Face++) generally demonstrate higher robustness than older open-source models (FaceNet, ArcFace).
• Occluding Attributes: Attributes that occlude the face, such as facemasks and sunglasses, universally degrade performance across all systems and demographics, though the extent of the drop varies.
• Demographic Sensitivity: Specific failure modes were isolated. For instance, all systems struggled with facial hair on female faces and thick beards on East Asian males, suggesting models struggle with out-of-distribution attribute-demographic combinations.
• Peripheral Attributes: Attributes on the periphery of the face (e.g., hair color, headbands, scarves) generally have negligible effects on performance, except for specific cases like shoulder-length hair on East Asian males.
• Ablation Studies:
  • Model Size: Larger models (ResNet-100) outperform smaller ones (ResNet-50) in generalizing to counterfactual changes.
  • Confounding Factors: Relaxing the "Specificity" criterion (allowing confounding factors) leads to significantly higher False Non-Match Rates (FNMR), proving the necessity of strict isolation of attributes for reliable evaluation.
  • Thresholds: Systems show improved robustness in utility-focused settings (FMR=1%) compared to security-focused settings (FMR=0.1%), though occluding attributes remain challenging.

Significance
The paper positions CounterFace as a tool for fine-grained diagnostics of FR systems. Unlike standard benchmarks that provide average accuracy, CounterFace enables the isolation of precise failure modes linked to specific attribute-demographic combinations. This allows model developers to identify exactly where and why a system fails (e.g., specific struggles with facial hair on certain demographics) and adjust their systems accordingly. The work demonstrates that automated, verified synthetic data can effectively replace costly human-in-the-loop processes for counterfactual evaluation, providing a scalable path toward more robust and fair biometric systems. The authors note that the dataset is gated and intended strictly for research evaluation, not training, to prevent misuse.