💻 computer science

Causal Supervision of Attention for Affective Behaviour Analysis

This paper proposes a causal supervision framework for attention mechanisms in affective behavior analysis that enforces subject-invariant focus, encourages non-redundant representations through cross-covariance regularization, and enhances feature expressiveness via gated nonlinear transformations to achieve state-of-the-art performance in the 11th Affective Behaviour Analysis in-the-wild Competition.

Original authors: Nemanja Rašajski, Konstantinos Makantasis, Antonios Liapis, Georgios N. Yannakakis

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nemanja Rašajski, Konstantinos Makantasis, Antonios Liapis, Georgios N. Yannakakis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to teach a robot to read your face and guess how you're feeling. You want it to spot a genuine smile or a frown, right? But here's the tricky part: the robot is easily distracted. It might think, "Oh, this person looks sad because they have a big nose," or "This person looks happy because the sun is shining on their forehead." These are like spurious correlations—clues that have nothing to do with actual feelings but happen to show up a lot in the training data. If the robot learns these bad habits, it will fail miserably when it meets someone new with a different nose or under different lighting.

The authors of this paper propose a clever new way to train the robot's "attention" so it stops looking at the wrong things and starts focusing on the real emotional signals, no matter who is in front of the camera. They call their method Causal Supervision of Attention.

The Three Magic Tricks

To fix the robot's bad habits, the team added three special ingredients to its learning process:

1. The "What If?" Game (Causal Supervision)
Usually, a robot just looks at a face and guesses. This team made the robot play a "What If?" game. They asked it: "If I forced you to ignore the nose and only look at the eyes, would your guess about the emotion change?"
They created a "counterfactual" version of the attention (a fake version where the robot looks at random spots) and compared it to the real one. Instead of just maximizing the raw difference in magnitude, they maximized the Total Direct Effect (TDE)—a specific measure of the causal contribution of attention to the prediction. By doing this, they encouraged the robot to focus on subject-invariant facial regions—areas that consistently predict emotion across different people, rather than just specific features of one person. This ensures the robot learns to attend to regions with invariant predictive value, effectively distinguishing true affective cues from spurious factors like identity or lighting, without necessarily ignoring identity entirely.

2. The "No Overlap" Rule (Cross-Covariance Regularization)
Think of the robot's brain as having two teams of workers: the Key team (who decide what to look at) and the Value team (who gather the details of what they see).
In standard models, these two teams often talk over each other and repeat the same information. It's like having two people in a room both shouting, "Look at the eyes!" while ignoring the mouth. The authors introduced a rule to make sure these two teams are complementary, not redundant. They added a penalty if the Key and Value teams started saying the same thing. This forces them to split the work: one might focus on the shape of the eyebrows, while the other focuses on the tension in the jaw, making the final picture much richer.

3. The "Super-Transformer" (SwiGLU)
Finally, the team upgraded the robot's brainpower. Standard models use a simple, straight-line math trick to process the details they've gathered. The authors swapped this out for a fancy, non-linear tool called SwiGLU.
Imagine the old method was a straight slide, and the new method is a twisting, turning rollercoaster. This allows the robot to capture much finer, more complex details about emotions that a simple slide would miss. It makes the robot's understanding of "fine-grained" feelings much sharper.

Did It Work?

The team tested this on a massive dataset of real-world videos called s-Aff-Wild2, where people are doing all sorts of things in natural environments. They measured success using a score called P, which adds up how well the robot did at three tasks: guessing Valence-Arousal (how positive/negative and how intense the feeling is), recognizing Expressions (like happy or sad), and spotting Action Units (tiny muscle movements in the face).

Here is what they found:

The Baseline: The standard robot (using a model called ConvNeXt) scored a 0.45 on the official validation set.
The New Method: When they used their new "Causal Supervision" tricks with a powerful visual encoder called FRoundation, the score jumped to 1.2214.
The Breakdown: Specifically, for guessing Valence-Arousal, they got a score of 0.5123. For recognizing expressions, they got an F1 score of 0.3116, and for spotting muscle movements, 0.3974.

They also ran the test five different times (called 5-fold cross-validation) to make sure the results weren't just luck. The new method consistently beat the old one, improving the average score from 0.8730 to 0.9569 with one type of brain (DINOv2) and from 1.0524 to 1.1948 with the other (FRoundation).

The Bottom Line

The paper suggests that by forcing the robot to distinguish true emotional cues from "fake" clues (like identity or lighting) and focus only on the parts of the face that have invariant predictive value across different subjects, we can build systems that understand human feelings much better. They didn't just guess this; they measured it across thousands of images and found that every single one of their three new tricks helped the robot perform better. While the robot isn't perfect yet (there's still room to improve those scores), this approach shows a clear path toward making machines that truly "get" us, rather than just guessing based on who we look like.

Technical Summary: Causal Supervision of Attention for Affective Behaviour Analysis

Problem Statement
Affective Behaviour Analysis (ABA) seeks to enable machines to infer human emotional states from behavioral signals, particularly facial expressions, in unconstrained ("in-the-wild") environments. While deep learning models have made significant progress, they often struggle with generalization to unseen subjects. This limitation arises because models frequently exploit spurious correlations—such as identity, pose, illumination, age, and demographic variations—rather than learning true, invariant affective cues. In the context of the 11th Affective Behaviour Analysis in-the-wild (ABAW) Competition, this challenge is formalized through a Multi-Task Learning (MTL) framework requiring simultaneous Valence-Arousal (VA) Estimation, Expression (EX) Recognition, and Action Unit (AU) Detection. The core difficulty lies in distinguishing emotion-relevant signals from these confounding factors to achieve robust subject-invariant representations.

Methodology
The authors propose a causally inspired attention pooling framework designed to improve both the selection and representation of affective cues. The architecture builds upon a standard pipeline involving a frozen Vision Transformer (ViT) backbone for feature extraction, followed by a Temporal Convolutional Network (TCN) for temporal modeling, and task-specific prediction heads. The framework introduces three specific modifications to the attention mechanism to enhance robustness and expressiveness:

Causal Supervision for Attention (CSA): To enforce attention on facial regions with invariant predictive value across subjects, the authors integrate a causal supervision signal. This involves a "factual" path where the model predicts based on learned attention weights, and a "counterfactual" path where attention weights are replaced by random, normalized vectors sampled from a Gaussian distribution. The model is trained to maximize the Total Direct Effect (TDE)—the difference between the factual and counterfactual predictions—thereby encouraging the attention mechanism to consistently identify causally relevant facial regions rather than relying on spurious correlations.
Cross-Covariance Independence Regularization: To address redundancy between the Key ( $K$ ) and Value ( $V$ ) projections in scaled dot-product attention, the authors introduce a cross-covariance regularization loss. By minimizing the Frobenius norm of the cross-covariance matrix between centered $K$ and $V$ representations, the method encourages complementary, non-redundant feature encoding, allowing more model capacity to be devoted to distinct, task-relevant information.
Nonlinear SwiGLU Value Projection: The standard linear projection for the Value ( $V$ ) component is replaced with a SwiGLU (Sigmoid Linear Unit Gated Linear Unit) transformation. This nonlinear layer increases the representational capacity of the value projection, enabling the model to capture finer-grained affective cues before the attention-based aggregation step.

Key Contributions
The paper presents a unified framework that integrates causal inference principles into attention mechanisms for affective computing. The primary contributions are:

The introduction of Causal Supervision for Attention, which utilizes counterfactual generation and TDE maximization to guide attention toward subject-invariant, emotion-relevant facial regions.
The formulation of a cross-covariance independence regularization loss to decouple Key and Value representations, reducing feature redundancy.
The replacement of linear Value projections with SwiGLU-based nonlinear transformations to enhance feature expressiveness.
A comprehensive evaluation on the s-Aff-Wild2 dataset demonstrating that these components collectively improve generalization across multiple affective tasks.

Experimental Results
The method was evaluated on the s-Aff-Wild2 dataset using two different backbones: DINOv2 (pre-trained on general-purpose images) and FRoundation (specialized for facial images). Performance was measured using the official ABAW MTL metric $P$ , which sums the mean Concordance Correlation Coefficient (CCC) for VA, the average F1 score for Expression ( $F_{EX}$ ), and the average F1 score for Action Units ( $F_{AU}$ ).

Baseline Comparison: Against the official baseline (ConvNeXt with MixAugment, $P=0.45$ ), the proposed method achieved significantly higher scores. With the FRoundation backbone, the method reached an overall score of 1.2214 on the official validation set ( $CCC_{VA}=0.5123$ , $F_{EX}=0.3116$ , $F_{AU}=0.3974$ ).
Cross-Validation: In a five-fold cross-validation protocol, the FRoundation backbone consistently outperformed DINOv2. The proposed method improved the mean multitask score $P$ from 1.0524 (baseline) to 1.1948 with FRoundation, and from 0.8730 to 0.9569 with DINOv2.
Ablation Study: The study confirmed that each component contributes to performance. While SwiGLU alone showed mixed results depending on the backbone, combining SwiGLU with either Causal Supervision or Covariance Regularization consistently improved performance. The full model (all three components active) yielded the best results, representing gains of approximately 9.6% (DINOv2) and 13.5% (FRoundation) over their respective baselines.

Significance
The paper claims that the proposed framework successfully addresses the challenge of spurious correlations in affective behaviour analysis by promoting subject-invariant attention. By explicitly modeling the causal contribution of attention and reducing redundancy in feature representations, the method produces more robust and expressive affective representations. The results suggest that integrating causal supervision and structural regularization into attention mechanisms is an effective strategy for improving generalization in unconstrained affective computing tasks, particularly when leveraging specialized facial backbones like FRoundation.

The Three Magic Tricks

Did It Work?

The Bottom Line

More like this