WIDER-FAIR: An Annotated Version of the WIDER-FACE Dataset for Fairness Evaluation

Technical Summary: WIDER-FAIR

Problem Statement
The deployment of face detection models in high-stakes domains (e.g., law enforcement, hiring) raises significant concerns regarding fairness and discriminatory outcomes. A primary obstacle to studying and mitigating these biases is the lack of face detection datasets that include annotations for sensitive features such as ethnicity and sex. While existing benchmarks like WIDER-FACE are widely used, they lack demographic information, preventing the evaluation of performance disparities across different groups. Furthermore, legislation such as the EU AI Act mandates the detection and mitigation of bias, necessitating datasets that align with legal definitions of protected classes (e.g., ethnic origin rather than just skin tone).

Methodology
To address this gap, the authors introduce WIDER-FAIR, a manually annotated subset of the WIDER-FACE dataset. The methodology involves the following steps:

1. Data Selection and Filtering: The authors selected images from the WIDER-FACE benchmark. To ensure annotation reliability and model detectability, they filtered out images containing faces with extreme occlusion, extreme blur, or extremely small bounding boxes (relative area < 0.15% of the image).
2. Annotation Process: A single human annotator manually labeled the perceived ethnicity and sex for each face.
   • Ethnicity Categories: Asian, Black, Indian, and White.
   • Sex Categories: Male and Female.
   • Exclusion Criteria: If a face was too difficult to classify or the sensitive value was unclear, it was labeled "Undetermined." If an image contained any such face, the entire image was discarded to avoid forcing uncertain labels.
   • Scope: The final dataset comprises 16,256 images covering the four ethnic groups and two sex categories.
3. Quality Assurance: The consistency of the annotations was validated using:
   • Face Embeddings: A face recognition model generated 128-dimensional embedding vectors.
   • Classification: A K-Nearest Neighbors (KNN) classifier was trained on these embeddings to predict sensitive features. High accuracy in this classification task suggested annotation coherence.
   • Visualization: t-SNE visualization demonstrated that faces with the same sensitive attributes clustered together in the embedding space.
4. Experimental Evaluation: The authors trained a YOLOv5 face detection model to evaluate the dataset's utility. They conducted ablation studies using a "Leave One Out" approach:
   • Leave One Ethnicity Out (LOEO): Training models while excluding one ethnic group from the training set.
   • Leave One Sex Out (LOSO): Training models while excluding one sex from the training set.
   • Metrics: Performance was measured using Recall at varying Intersection over Union (IoU) thresholds. Fairness was evaluated using Equal Opportunity, quantified by the disparity (standard deviation) of recall across sensitive groups.

Key Results

• Annotation Quality: The KNN classifier achieved high accuracy in predicting the annotated sensitive features, and t-SNE visualizations confirmed that groups formed distinct clusters, indicating the annotations are coherent.
• Baseline Performance Disparities: When trained on the full dataset, the model exhibited lower detection performance (recall) for Black individuals compared to Asian, Indian, and White individuals. White individuals showed slightly lower performance than Asian and Indian groups. A smaller performance gap was observed between sexes, with male faces being slightly harder to detect than female faces.
• Ablation Findings:
  • Ethnicity: Excluding the Black group from training resulted in the most significant drop in performance for that specific group and caused the largest increase in disparity compared to other LOEO scenarios. This suggests that the presence of Black faces in the training data is critical for the model's generalization and fairness.
  • Sex: Excluding female faces reduced the disparity but lowered the model's ability to detect females. Excluding male faces increased disparity, suggesting male faces are also important for generalization.

Significance and Claims
The paper claims that WIDER-FAIR provides a necessary resource for the community to evaluate and understand bias in face detection models, building upon a well-known benchmark. The authors emphasize that the dataset is intended to support fairness evaluation tasks by providing demographic annotations aligned with legal definitions of protected features.

The authors maintain a modest tone regarding the dataset's limitations, explicitly stating:

• The annotations were performed by a single human based on perceived ethnicity and sex, not ground-truth self-identification.
• The dataset cannot be guaranteed to be free of unintended bias or annotation errors.
• The exclusion of difficult images (blur/occlusion) involved arbitrary thresholds.

Consequently, the paper positions WIDER-FAIR as a tool to initiate research into fairness in face detection, encouraging future researchers to treat the dataset with appropriate caution while leveraging its potential to identify performance disparities, such as the notable disadvantage faced by Black individuals in current detection models.