Benchmarking Face Recognition without Real Faces

Technical Summary: Benchmarking Face Recognition without Real Faces

Problem Statement
While synthetic face datasets have advanced to the point where models trained entirely on artificial data achieve recognition accuracy rivaling those trained on real photographs, the evaluation pipeline remains dependent on real-world biometric data. Current research practices require the use of real-face benchmarks (e.g., LFW, IJB-C) to validate models trained on synthetic data. This reliance leaves the privacy and ethical burdens of biometric data collection only partially solved, as the evaluation phase still necessitates access to sensitive real-world identities. Furthermore, existing research on synthetic benchmark datasets is limited and lacks clear guidance on their practical applicability, raising concerns about distributional mismatches, poor identity separability, and demographic biases in synthetic data.

Methodology
The authors conducted a comprehensive empirical evaluation to determine if synthetic datasets can reliably replace real benchmarks. The study involved:

• Datasets: A comparison of 12 synthetic face recognition datasets (generated via GANs, Diffusion Models, and hybrid 3D morphable approaches) against 7 established real benchmarks (LFW, CPLFW, CALFW, AgeDB-30, CFP-FP, IJB-B, and IJB-C).
• Models: Evaluation across 24 pre-trained face recognition models, encompassing both Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). Notably, the study included models trained entirely on synthetic data.
• Metrics: The evaluation utilized standard biometric verification metrics, including recognition accuracy, Equal Error Rate (EER), False Match Rate (FMR) at various thresholds (FMR100, FMR1000), ZeroFMR, and True Acceptance Rate (TAR).
• Statistical Analysis: To assess benchmarking fidelity, the authors measured:
  • Linear Agreement: Pearson's correlation coefficient ($r$) between scores on synthetic and real datasets.
  • Ranking Consistency: Spearman's rank correlation ($\rho$) to determine if the relative ordering of models is preserved.
  • Distributional Properties: Fréchet Inception Distance (FID) to quantify the domain gap between synthetic and real distributions, and analysis of similarity score distributions for mated vs. non-mated pairs.
• Protocols: The study adhered to the specific evaluation protocols of each real dataset (e.g., Unrestricted with Labeled Outside Data for LFW, template-based for IJB-B/C) and applied a consistent 3,000 mated/3,000 non-mated pair protocol for synthetic datasets.

Key Contributions

1. First Comprehensive Benchmarking Study: The paper provides the first systematic evaluation of 12 synthetic datasets as benchmarks across 24 diverse models, comparing them directly against 7 real-world standards.
2. Performance Breakdown: It offers a detailed analysis of benchmarking performance across recognition accuracy and multiple standard biometric metrics (EER, FMR, ZeroFMR).
3. Identification of Viable Substitutes: The study identifies MorphFace and Vec2Face as the strongest synthetic candidates, demonstrating they can serve as reliable substitutes for real benchmarks.
4. Factor Analysis: The authors investigate the underlying factors influencing benchmarking utility, specifically linking performance to domain gap (FID), identity separability, and intra-class diversity.
5. Reproducibility: The code and evaluation protocols are made publicly available to promote reproducible research.

Results

• Variable Fidelity: Benchmarking fidelity varies significantly across synthetic candidates. While some datasets (e.g., ControlFace10k, SynMulti-PIE) showed negative or weak correlations with real benchmarks, others performed exceptionally well.
• Top Performers: MorphFace and Vec2Face achieved the highest alignment with real benchmarks. Their Pearson's $r$ and Spearman's $\rho$ values fell within the natural disagreement levels observed among the real benchmarks themselves.
  • For Mean Pearson's $r$, both achieved 0.90.
  • For Mean Spearman's $\rho$, both achieved 0.87.
  • Their 95% confidence intervals overlapped with the agreement levels of real-real dataset pairs.
• Metric Sensitivity: Synthetic datasets generally showed stronger correlations under EER and weaker correlations under ZeroFMR. The ZeroFMR gap was attributed to unnatural distributions of outliers (highly similar non-mated pairs) in some synthetic sets.
• Distributional Insights:
  • Low FID scores were necessary but not sufficient for good benchmarking performance. For instance, CemiFace had a lower FID than MorphFace and Vec2Face but performed worse as a benchmark.
  • Successful synthetic datasets (MorphFace, Vec2Face) exhibited similarity score distributions that closely resembled those of challenging real datasets (like AgeDB-30 and CFP-FP), indicating sufficient intra-class diversity and clear identity separability.
  • Poor performers like ControlFace10k suffered from low between-class variability and poor identity separability, leading to high false match rates among non-mated pairs.
• Training vs. Benchmarking: A correlation was observed where datasets performing well for training also tended to perform well for benchmarking, though the relationship is not strictly one-to-one (e.g., SynFace was weak for training but relatively stronger for benchmarking).

Significance and Claims
The paper claims that well-constructed synthetic datasets can support reliable comparative evaluation for face recognition. The results demonstrate that it is possible to move the field closer to a fully synthetic and privacy-preserving pipeline for both training and benchmarking.

The authors assert that:

• Synthetic benchmarks are not merely academic exercises; they can practically replace real data for model evaluation if they meet specific criteria.
• The primary determinants of benchmarking quality are data diversity (intra-class variation) and identity separability (clear distinction between genuine and impostor pairs).
• Future synthetic datasets intended for benchmarking should empirically demonstrate alignment with real benchmarks, ensuring they do not exceed the natural disagreement levels found among real datasets and maintain agreement at critical operating points (EER, FMR100, FMR1000).

The study concludes that while not all synthetic data is suitable, specific high-fidelity datasets like MorphFace and Vec2Face have reached a maturity level where they can effectively reduce reliance on privacy-sensitive real facial data for the evaluation of face recognition systems.