Technical Summary: SIG – A Synthetic Identity Generation Pipeline

Problem Statement

The evaluation of face recognition systems faces significant hurdles regarding data availability, ethics, and representativeness. High-quality evaluation datasets must be disjoint from training data, ethically sourced, and balanced across demographic attributes (race, gender, age) and imaging conditions (pose, lighting). However, existing approaches are flawed:

• Internet Scraping: Collecting data from the web often lacks user consent, raising ethical concerns and leading to the withdrawal of major datasets (e.g., MS-Celeb-1M) due to privacy violations.
• Controlled Collection: Gathering data in controlled environments with consent is time-consuming, expensive, and logistically difficult, creating barriers for researchers lacking immense resources.
• Existing Datasets: Current public datasets (e.g., LFW, VGGFace2, BUPT-Balancedface) often suffer from imbalances in demographics, lack of pose variance, or insufficient control over specific attributes like age progression and lighting. Furthermore, most lack comprehensive, simultaneous labeling and control for race, gender, pose, and age.
• Synthetic Data Limitations: While synthetic data offers a solution, existing generation methods (GANs, diffusion models) often lack fine-grained control over demographic attributes without specific, resource-intensive retraining. They frequently fail to preserve identity consistency across different poses or lack mechanisms to control race, gender, and age simultaneously.

Methodology: The Synthetic Identity Generation (SIG) Pipeline

To address these gaps, the authors propose the Synthetic Identity Generation (SIG) pipeline, a system built on Stable Diffusion (specifically the "Absolute Reality" fine-tune) enhanced with ControlNets. The pipeline is designed to generate high-quality, hyper-realistic synthetic identities with precise control over identity features and pose.

1. Architecture

The SIG pipeline consists of two primary subsystems:

• Prompt Builder: This system constructs the text prompts required to guide image generation.
  • Name Generation: It utilizes GPT-4 to compile a diverse list of 15,900 names across 139 countries, distributed among four racial groups (African, Asian, Caucasian, Indian) and genders.
  • Keyword Blending: To create unique identities, the system employs a "keyword blending" technique using the syntax [Name 1|Name 2|Name 3]. This associates a unique name triplet with a "synthetic identity," theoretically allowing for over 10 billion unique combinations per race group. This technique encourages the model to merge facial features associated with the specific names, broadening the diversity of generated faces.
  • Attribute Control: Prompts are meticulously crafted to include specific attributes for race, background, hairstyle, facial expression, age, and gender.
• Image Generator: This component consumes the prompts to synthesize images.
  • Core Model: It utilizes the StableDiffusionControlNetPipeline from Hugging Face's Diffusers library.
  • Pose Control: The pipeline integrates the OpenPose ControlNet. It extracts pose information (skeleton/landmarks) from a reference image featuring a person in various orientations. This pose data is then used as a conditioning vector to guide the generation of synthetic identities, ensuring specific pose requirements (e.g., right-facing, front-facing, left-facing) are met while maintaining identity consistency.
  • Identity Consistency: By using the same prompt (name triplet) and conditioning the pose via ControlNet, the system generates multiple images of the same synthetic identity across different poses.

Key Contributions

The paper presents three primary contributions:

1. The SIG Pipeline: A demonstration of a pipeline capable of generating high-quality synthetic identities with controllable pose, race, gender, and age without requiring specific retraining of the underlying diffusion model for each attribute.
2. ControlFace10k Dataset: The release of an open-source evaluation dataset containing 10,008 images representing 3,336 unique synthetic identities. The dataset is strictly balanced across:
   • Race: 834 identities each for African, Asian, Caucasian, and Indian.
   • Gender: 1,668 male and 1,668 female identities.
   • Age: Identities are distributed across three age groups: 25, 50, and 65 years.
   • Pose: Each identity includes images in right-facing, front-facing, and left-facing poses.
3. Empirical Analysis: An evaluation of ControlFace10k using state-of-the-art face recognition algorithms (ArcFace and GhostFaceNet) to compare similarity score distributions against a real-world dataset (BUPT-Balancedface).

Results and Analysis

The authors analyzed ControlFace10k to validate its utility as an evaluation tool:

• Similarity Score Distributions: When comparing non-mated pairs (different identities) across race groups, the similarity scores for ControlFace10k generally aligned with the real-world BUPT dataset, centering around 0.5 (indicating orthogonal embeddings).
  • Model-Dependent Bias: The analysis revealed a "model-effect." ArcFace showed a slight shift in distributions for certain groups (e.g., African and Indian identities had higher similarity scores compared to BUPT), suggesting potential biases in the model's training data or the generation process. GhostFaceNet's distributions overlapped more closely with BUPT, indicating potentially better distinctiveness for synthetic identities.
  • Asian Group Overlap: Both models showed significant overlap with BUPT scores for the Asian group, which the authors hypothesize could stem from biases in the training data of the recognition models or the generation capabilities of the SIG pipeline for this specific demographic.
• Pose Invariance: The study compared similarity scores for frontal poses versus varied poses (front, left, right). Results indicated that while models perform well on frontal faces, their consistency drops when comparing varied poses of the same synthetic identity. This highlights the continued challenge of pose-invariant face recognition, even with high-quality synthetic data.
• Identity Consistency: The "SIG Frontal Sample" (3,360 identities with three frontal images each) demonstrated high similarity scores, confirming that the pipeline successfully maintains consistent facial attributes for a single synthetic identity.

Significance and Claims

The paper positions SIG and ControlFace10k as a critical step toward ethical, balanced, and controllable evaluation of face recognition systems.

• Ethical Compliance: The pipeline eliminates the need for internet scraping or costly controlled data collection, generating data that is ethically sourced by design.
• Bias Assessment: By providing a dataset with precise control over race, gender, age, and pose, SIG enables researchers to systematically assess algorithmic bias and fairness across diverse demographic groups, a capability lacking in current public datasets.
• Privacy and Security: Synthetic data mimics real-world distributions while ensuring individual privacy is maintained, addressing legal and ethical challenges associated with real data usage.
• Future Utility: The authors claim that ControlFace10k serves as a foundation for future research, allowing for the simulation of edge cases (e.g., specific age progressions or rare poses) that are difficult to capture in the real world. They emphasize that the dataset is intended for evaluation, not training, to ensure disjoint testing conditions.

The authors conclude that while their current focus is on controlling critical demographic attributes, the framework is scalable and extensible, offering a pathway to develop more robust, unbiased, and generalizable face recognition systems.