Solution to the 10th ABAW Expression Recognition Challenge: A Robust Multimodal Framework with Safe Cross-Attention and Modality Dropout

This paper presents a robust multimodal framework for the 10th ABAW Expression Recognition Challenge that utilizes a dual-branch Transformer with safe cross-attention and modality dropout to dynamically fuse audio and visual data, effectively addressing partial occlusions, missing modalities, and class imbalance to achieve 60.79% accuracy on the Aff-Wild2 validation set.

The Gist

Imagine you are trying to guess what a friend is feeling just by looking at them and listening to their voice. Usually, this is easy. But what if they are standing in the dark? What if they turn their back to you? What if they are shouting over a loud construction site?

This is the exact problem the researchers in this paper are trying to solve. They built a "super-smart computer brain" to recognize human emotions in the messy, chaotic real world, specifically for the 10th ABAW Challenge.

Here is how their solution works, explained with some everyday analogies:

1. The Two-Eyed, Two-Eared Detective

Most emotion detectors just look at faces (like a security camera). But in the real world, faces get blocked by hands, hats, or bad lighting.

The team built a two-branch detective:

• The Visual Branch (The Eyes): Uses a super-advanced AI (called BEiT-large) to read facial expressions.
• The Audio Branch (The Ears): Uses a different AI (called WavLM-large) to listen to tone of voice, pitch, and speed.

The Magic Trick: Usually, if you can't see someone's face, you can't guess their emotion. But this system is designed so that if the "eyes" go blind, the "ears" take over immediately. It's like a car with a backup steering wheel; if the main one jams, the backup kicks in so you don't crash.

2. The "Safe" Handshake (Safe Cross-Attention)

In computer science, these two branches need to talk to each other to combine their clues. This is called "Cross-Attention."

Imagine the Visual branch and Audio branch are two people shaking hands to share information.

• The Problem: If the Visual branch is looking at a black screen (because the person walked out of the frame), it tries to shake hands with nothing. In normal computers, this causes a glitch or a crash.
• The Solution: The team invented a "Safe Handshake." If the Visual branch has no data, the system has a special rule: "Don't panic. Just let the Audio branch speak for itself." It mathematically protects the system from breaking when data is missing, ensuring the computer doesn't get confused by silence or darkness.

3. The "Training with Blindfolds" (Modality Dropout)

How do you teach a student to swim if you only let them practice in a pool with perfect water? They will fail when they hit a wave.

To make their AI robust, the researchers used a technique called Modality Dropout. During training, they randomly put "blindfolds" on the Visual branch.

• The Analogy: Imagine training a detective by covering their eyes 10% of the time. They are forced to learn how to solve the case using only their ears.
• The Result: When the AI is tested in the real world and the camera gets blocked, it doesn't freeze. It's already practiced for that scenario.

4. Fighting the "Popular Vote" Bias (Focal Loss)

In the dataset they used (Aff-Wild2), there are way more videos of people looking "Neutral" or "Happy" than there are of people looking "Disgusted" or "Fearful."

• The Problem: A normal AI is lazy. It learns that if it just guesses "Happy" every time, it will be right most of the time. It ignores the rare, difficult emotions.
• The Solution: They used a special scoring system called Focal Loss.
• The Analogy: Imagine a teacher grading a test. If a student gets the easy questions right, they get a tiny point. But if they get the hard, rare questions right, they get a massive bonus point. This forces the AI to stop being lazy and actually study the difficult, rare emotions.

5. The "Smooth Movie" Editor (Sliding Window & Soft Voting)

Emotions don't switch instantly like a light switch. They flow like a river. If you look at a video frame-by-frame, the AI might get jittery: Happy, Sad, Happy, Sad—all in one second. That looks weird.

• The Solution: They use a Sliding Window. Instead of judging one single frame, they look at a chunk of the video (like a 2-second clip) and average the results.
• The Analogy: It's like watching a movie and smoothing out the shaky camera work. If the AI thinks someone is "Angry" for one frame but "Neutral" for the next, the system averages them out to say, "They are probably getting annoyed," rather than flipping back and forth wildly. This makes the final result feel natural and steady.

The Result

By combining these tricks—listening when sight fails, training with blindfolds, rewarding the AI for solving hard puzzles, and smoothing out the final movie—the team achieved a 60.79% accuracy on a very difficult test.

In short: They built an emotion detector that doesn't just look at faces; it listens, it adapts when things go wrong, and it knows how to handle the messy reality of the real world.

Technical Summary

Here is a detailed technical summary of the paper "Solution to the 10th ABAW Expression Recognition Challenge: A Robust Multimodal Framework with Safe Cross-Attention and Modality Dropout."

1. Problem Statement

The paper addresses the challenges of Emotion Recognition in-the-wild (EXPR), specifically within the context of the 10th ABAW (Affective Behavior Analysis in-the-wild) competition. The core difficulties identified are:

• Unconstrained Environments: Real-world data suffers from adverse lighting, varying head poses, and cultural differences.
• Missing Modalities: Subjects frequently experience partial occlusions or leave the camera's field of view, leading to a complete absence of visual data.
• Severe Class Imbalance: The Aff-Wild2 dataset exhibits a long-tail distribution where certain emotions are rare, causing standard models to bias toward majority classes.
• Temporal Dynamics: Emotions transition continuously, and frame-level classification often suffers from jitter (instability) due to noise.

2. Methodology

The authors propose a robust, end-to-end multimodal framework that integrates visual and audio data using a Dual-Branch Transformer architecture. The methodology consists of five key components:

A. Feature Extraction and Pre-training

• Visual Modality: Uses BEiT-large as the backbone. To improve generalization, the model undergoes a two-stage training process:
  1. Pre-training on a large-scale mixed static dataset (Raf-DB, FERPlus, AffectNet) covering basic emotions and neutral states.
  2. Domain-adaptive fine-tuning on the target Aff-Wild2 video frames.
• Audio Modality: Uses WavLM-large pre-trained on a large speech corpus to capture acoustic prosody and subtle emotional fluctuations. Audio features are linearly interpolated to align temporally with video frames.

B. Multimodal Attention Network

The core architecture employs a dual-branch Transformer with:

• Cross-Attention Mechanisms: Features from both branches ($H_v$ and $H_a$) are processed through bidirectional cross-attention blocks ($H_{v \to a}$ and $H_{a \to v}$) to facilitate inter-modality interaction.
• Learnable Gating Fusion: A gating mechanism ($G_v$) dynamically regulates the fusion of unimodal context and cross-modal features, allowing the network to adaptively balance contributions based on input quality.

C. Safe Cross-Attention and Modality Dropout

To handle missing visual data (e.g., occlusion), the authors introduce two specific mechanisms:

• Modality Dropout: During training, visual inputs are randomly masked with probability $p$ to simulate real-world signal loss, preventing over-reliance on vision.
• Safe Attention Logic: If a window lacks visual features entirely, the system prevents the softmax function from generating invalid values. It temporarily unmask the first token to allow the forward pass, then manually sets the attention output to zero. Combined with residual connections, this allows the network to gracefully degrade to rely entirely on the audio branch without crashing or producing garbage predictions.

D. Optimization Objective

• Focal Loss: To address the long-tail distribution, the framework replaces standard cross-entropy with Focal Loss ($\gamma=2.0$). This down-weights easy, high-frequency samples, forcing the model to focus on hard, minority-class examples.
• Invalid Frame Handling: Frames labeled as invalid ($-1$) are explicitly ignored during loss computation to prevent gradient noise.

E. Inference Strategy

• Sliding Window & Soft Voting: Instead of hard label voting, the model uses overlapping sliding windows ($W=64, S=8$) to perform local spatiotemporal modeling.
• Logit Averaging: Predictions for a specific frame are averaged across all windows covering it to smooth transitions.
• Median Filtering: A final median filter ($k=11$) is applied to reduce frame-level classification jitter while preserving emotional state boundaries.

3. Key Contributions

1. Safe Cross-Attention Mechanism: A novel architectural design that ensures system stability and maintains classification performance even when visual modalities are completely absent, a common failure point in previous multimodal systems.
2. Modality Dropout Strategy: A training technique that explicitly simulates missing data, forcing the model to learn robust audio-based fallback strategies.
3. Dynamic Fusion with Gating: An adaptive fusion mechanism that balances visual dominance (which is generally stronger) with audio supplementation, rather than using static weights.
4. Comprehensive Optimization: The combination of Focal Loss for class imbalance and sliding-window soft voting for temporal smoothing specifically tailored to the Aff-Wild2 dataset characteristics.

4. Experimental Results

The framework was evaluated on the Aff-Wild2 validation set:

• Performance: Achieved an Accuracy of 60.79% and an F1-score of 0.5029.
• Ablation Studies:
  • Modality Dropout: Setting the dropout probability to $p=0.10$ yielded the best results (F1: 0.5029), significantly outperforming the no-dropout baseline (F1: 0.4764). Higher dropout rates ($>0.15$) degraded performance due to excessive loss of primary visual information.
  • Network Capacity: A medium-capacity network ($d=256$, $l=3$ layers) outperformed larger configurations ($d=512$ or $l=4$), indicating that highly parameterized models overfit the noisy, limited-scale Aff-Wild2 data.
  • Fusion Weights: While visual features are dominant, a fusion weight of $\lambda=0.7$ (visual) outperformed vision-only ($\lambda=1.0$), proving the necessity of audio as a supplementary cue.

5. Significance

This work provides a critical solution for deploying emotion recognition systems in real-world, uncontrolled environments. By addressing the robustness gap caused by missing modalities and the bias gap caused by class imbalance, the proposed framework moves beyond the limitations of static, controlled-dataset models. The "Safe Attention" mechanism is particularly significant as it offers a practical blueprint for building fault-tolerant AI systems that can operate reliably even when sensor data is compromised, a prerequisite for applications in mental health monitoring, social robotics, and personalized education.