The Circumplex Degeneracy Behind the Rare-Class Limit in Affect Recognition

Technical Summary: The Circumplex Degeneracy Behind the Rare-Class Limit in Affect Recognition

Problem Statement
In-the-wild facial expression recognition systems consistently achieve usable accuracy on common emotions but fail persistently on a few rare classes, specifically anger and fear (on Aff-Wild2) or anger and contempt (on AffectNet). The standard interpretation attributes this failure to class imbalance, suggesting solutions like re-weighting, balanced sampling, or focal losses. However, these frequency-based interventions fail to improve performance on the hardest classes. This paper posits that the failure is not a matter of frequency but of affect geometry: the rare classes are geometrically degenerate (nearly coincident) on Russell's circumplex model of emotion, a property that bounds the effectiveness of any loss function or cost design.

Methodology
The authors conduct a controlled multi-task study on two benchmarks (Aff-Wild2 and AffectNet) using a shared encoder with task-specific heads for Valence-Arousal (VA), Expression (EXPR), and Action Units (AU).

1. Circumplex-Cost Optimal Transport (OT) Term: The primary instrument is a structured loss term that prices expression confusions based on their Euclidean distance in the valence–arousal plane. The ground metric is derived directly from the dataset's own VA annotations, computing class centroids ($\mu_c$) and using the distance $D_{cc'} = \|\mu_c - \mu_{c'}\|$ as the transport cost.
2. Controlled Comparisons: To isolate whether performance gains stem from the specific affect geometry or merely from the presence of a non-trivial penalty, the authors introduce two critical controls rarely used in this literature:
   • Uniform Cost: Replaces the structured circumplex distances with a constant value, reducing the OT term to a generic confidence penalty.
   • Permuted Cost: Shuffles class indices to preserve the multiset of pairwise distances but destroys the semantic geometry.
3. Error-Structure Metrics: Since standard Macro-F1 is invariant to which wrong class is predicted, the authors introduce geometry-aware metrics: Error Severity (mean circumplex distance of misclassifications), Quadrant Preservation, and Affect-Weighted Accuracy.
4. Representational Analysis: The study compares face self-supervised encoders (FSFM) against face vision-language encoders (FaRL) and general vision-language models (SigLIP 2) to determine if the feature space already encodes the circumplex structure.
5. Action Unit (AU) Augmentation: An experiment constructs a ground cost using AU signatures to explicitly separate the degenerate pairs (e.g., anger vs. fear), testing if a cost with higher geometric resolution can overcome the representation's limitations.

Key Results

• Score Gain is Generic, Not Geometric: The circumplex-cost OT term improves the official multi-task score (PMTL) and Expression Macro-F1 on both datasets. However, the uniform cost control reproduces these gains entirely. On Aff-Wild2, the uniform cost is statistically indistinguishable from the circumplex cost ($p=0.625$); on AffectNet, the uniform cost significantly outperforms the circumplex cost ($+0.057$ over base vs. $+0.052$ for circumplex). This indicates the score improvement is an artifact of adding a generic confidence penalty, not the affect geometry itself.
• Error Shaping is Dataset-Dependent: While the score gain is generic, the circumplex geometry does reshape the error structure on Aff-Wild2, making surviving mistakes affectively nearer the truth (lower error severity, $p=0.031$). This effect does not survive on AffectNet, where a visual confound (contempt mistaken for happiness) overwhelms the geometric signal.
• The Rare-Class Limit is Stable: The degenerate pairs (anger–fear on Aff-Wild2; anger–contempt on AffectNet) resist all interventions. Frequency-based losses, the transport term, and even an AU-augmented cost (which successfully separates the pairs in the cost matrix) fail to recover the F1 scores of these classes.
• Representational Origin: The failure is identified as representational. Vision-language encoders (FaRL) already encode the circumplex structure in their features, making the geometric loss term redundant. Conversely, face self-supervised encoders (FSFM) lack this alignment. Crucially, even when the cost geometry is made perfectly separable (via AU signatures), the model cannot separate the classes, proving the bottleneck lies in the feature representation, not the loss weighting.

Key Contributions

1. Isolation of Geometric Effects: The paper demonstrates that apparent benefits of affect-geometry supervision are often generic regularization effects. It establishes shuffled and uniform-cost controls as necessary standards for evaluating structured-cost losses in affect recognition.
2. Diagnosis of Rare-Class Failure: The authors diagnose the persistent failure of rare expressions as a circumplex degeneracy. This degeneracy is a representational limit; no cost design can manufacture separability that the feature representation lacks.
3. New Metrics and Controls: The paper introduces error severity, quadrant preservation, and the shuffled-cost control to distinguish between geometric error shaping and generic regularization, providing an honest account of where geometric effects appear and where they do not.
4. Representational Analysis: Through controlled comparisons of pretraining domains, the study shows that the ability of a model to utilize affect geometry depends on whether the pretraining domain (e.g., vision-language vs. self-supervised) has already encoded that structure.

Significance and Claims
The paper concludes that progress on rare expressions cannot be achieved by "repricing" confusions through better loss functions or cost geometries. Instead, the solution requires representations that distinguish the classes. The authors argue that the field has over-attributed success to geometric supervision when it is often a generic penalty effect, and that the true barrier to recognizing rare emotions is the collapse of these classes in the feature space due to their geometric degeneracy. The study advocates for shifting focus from loss engineering to developing feature representations capable of resolving these degenerate pairs, potentially by fusing orthogonal signals (like Action Units) at the feature level rather than the loss level.