AffectNet+: A Database for Enhancing Facial Expression Recognition with Soft-Labels

Technical Summary: AffectNet+: A Database for Enhancing Facial Expression Recognition with Soft-Labels

Problem Statement
Automated Facial Expression Recognition (FER) faces significant challenges due to intra-class variations (diverse manifestations of a single emotion) and inter-class similarities (overlapping facial muscle activations across different emotions). These issues are exacerbated by "compound expressions," where a single facial image conveys a mixture of emotions rather than a discrete category. Existing FER datasets, such as AffectNet, rely on "hard-labels," assigning a single discrete emotion category to each image. This approach ignores the nuance of mixed emotions and the uncertainty inherent in human annotation, often leading to noisy labels and biased models. Furthermore, current datasets frequently lack comprehensive metadata (e.g., age, gender, ethnicity, head pose) and fail to account for the varying difficulty levels of recognizing expressions in "in-the-wild" settings.

Methodology
The authors propose a novel framework termed Soft-FER, which replaces discrete hard-labels with soft-labels. A soft-label is defined as a vector of probability scores representing the extent to which multiple emotion categories are simultaneously present in a single facial image. To generate these labels for the AffectNet+ dataset, the authors utilized a Multi-Annotated Set (MAS) containing 36,000 images labeled by at least two human annotators to ensure higher reliability.

The soft-label generation process employs two distinct deep learning methods:

1. Ensemble of Binary Classifiers (EBC): This approach trains eight binary classifiers (one for each emotion class in a one-vs-rest manner) using an ensemble of three different architectures: ResNet-50, EfficientNet-B3, and XceptionNet. To handle class imbalance during training, the authors proposed a specific negative sampling strategy based on the correlation of Action Units (AUs) between emotions. The final semantic score for each emotion is calculated by aggregating the probability outputs of the three classifiers, weighted by their respective confidence scores (average accuracy).
2. Action Unit (AU)-Based Classifier: This method leverages the Emotional Facial Action Coding System (EMFACS). A multi-head ResNet-50 model is trained simultaneously to perform binary classification and to predict a 21-dimensional AU representation vector. The model learns the specific AU combinations associated with each emotion. The final probability is derived by combining the binary classification output with a similarity score between the predicted AU vector and the ground-truth AU vector for each emotion class.

The final soft-label vector for each image is the average of the semantic scores produced by the EBC and the AU-based classifier.

Based on the alignment between the generated soft-labels and the original human-assigned hard-labels, the AffectNet+ dataset is partitioned into three subsets:

• Easy: The hard-label matches the emotion with the highest probability in the soft-label vector.
• Challenging: The hard-label falls within the top three probabilities of the soft-label vector but is not the highest.
• Difficult: The hard-label does not appear in the top three probabilities of the soft-label vector.

Additionally, the authors updated the metadata for the dataset, adding age, gender, ethnicity, head pose, and two sets of facial landmarks (68-point and 28-point) using pre-trained deep learning models.

Key Contributions

• Introduction of Soft-Labels: The paper introduces the concept of soft-labels for facial expression datasets, providing a more realistic, multi-dimensional description of facial expressions that captures compound emotions and varying intensities.
• AffectNet+ Dataset: The authors present AffectNet+, a next-generation dataset built upon AffectNet. It includes soft-labels for all images, three difficulty-based subsets (Easy, Challenging, Difficult), and enriched metadata (demographics, pose, landmarks).
• Novel Labeling Methodology: The paper proposes and validates two automatic methods (EBC and AU-based classifier) for calculating soft-labels, addressing the high cost and time required for manual multi-annotation.
• Subjective Evaluation: A human study involving six participants demonstrated that soft-labels are preferred over hard-labels for describing facial expressions and that humans can reliably distinguish accurate soft-labels from random ones.

Experimental Results

• Label Generation: The EBC model achieved high average accuracies (ranging from ~78% to 93% depending on the ensemble configuration) across all emotion classes, with the ensemble significantly reducing standard deviation compared to individual models. The AU-based classifier showed strong performance for most emotions, though it struggled with Fear and Anger due to high AU overlap.
• Baseline Performance (Hard-FER): A baseline ResNet-50 model trained on hard-labels achieved 85.86% accuracy on the "Easy" subset but dropped significantly to 51.62% on "Challenging" and 34.34% on "Difficult" subsets. The model exhibited high confusion between similar emotions (e.g., Contempt vs. Neutral/Happy, Disgust vs. Anger).
• Baseline Performance (Soft-FER): When evaluated using soft-labels, the baseline model showed lower Weighted Mean Absolute Error (W-MAE) and Weighted Failure Rate (W-FR) on the Easy subset compared to the Difficult subset. Notably, Soft-FER reduced the confusion between Neutral and Contempt compared to Hard-FER, as it treats each emotion class independently rather than forcing a single discrete choice.
• Human Evaluation: In subjective tests, 65% of participants preferred soft-labels over hard-labels for describing images, and 81% could correctly identify accurate soft-labels against random alternatives.

Significance and Claims
The paper claims that AffectNet+ addresses the limitations of existing FER datasets by providing a more nuanced representation of facial expressions through soft-labels. By acknowledging that facial expressions often involve compound emotions and varying intensities, the dataset allows for the development of models with smoother decision boundaries. The authors posit that AffectNet+ will facilitate research in several areas, including:

• Quantifying uncertainty in FER.
• Developing multi-labeling models capable of predicting multiple expressions simultaneously.
• Mitigating the effects of noisy labels and imbalanced data distributions.
• Improving model interpretability and generalization in "in-the-wild" settings.
• Enabling domain adaptation and self-supervised learning through enriched metadata.

The authors conclude that AffectNet+ represents a significant step forward in affective computing, offering a resource that better reflects the complexity of human emotional expression than previous hard-label datasets.