💻 computer science

Hierarchical Granularity Alignment and State Space Modeling for Robust Multimodal AU Detection in the Wild

This paper proposes a novel multimodal framework for robust in-the-wild facial Action Unit detection that integrates foundation models (DINOv2 and WavLM) with a Hierarchical Granularity Alignment module and a Vision-Mamba architecture to effectively capture fine-grained semantic shifts and ultra-long temporal dependencies, achieving state-of-the-art performance on the Aff-Wild2 dataset.

Original authors: Jun Yu, Yunxiang Zhang, Naixiang Zheng, Lingsi Zhu, Guoyuan Wang

Published 2026-03-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jun Yu, Yunxiang Zhang, Naixiang Zheng, Lingsi Zhu, Guoyuan Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a friend's emotions just by watching a video of them talking in a crowded, noisy park. This is the challenge of Facial Action Unit (AU) detection. An "Action Unit" is just a fancy term for a tiny muscle movement, like a slight twitch of the eyebrow or a small pull of the lip corner. These tiny movements tell us if someone is happy, sad, or angry.

Doing this in a controlled lab is easy. But doing it "in the wild" (like in that noisy park) is a nightmare because of bad lighting, weird head angles, and background noise.

This paper presents a new, super-smart system to solve this problem. Here is how it works, broken down into simple concepts:

1. The Problem: Old Tools Were Too Clumsy

Previous AI systems were like trying to read a fine-print menu through thick, foggy glasses.

The Eyes: Old cameras (visual encoders) were too zoomed out. They could see the whole face but missed the tiny muscle twitches.
The Ears: Old microphones (audio models) were too focused on what people said (the words) and ignored how they said it (the sighs, the shaky voice, the breathing).
The Memory: Old systems had short attention spans. They could only remember the last few seconds of a video, missing the long story of how an emotion built up over time.

2. The Solution: A New Team of Experts

The authors built a new system that acts like a team of specialized detectives, using three main upgrades:

A. The "Super-Sharp" Eyes and Ears (Foundation Models)

Instead of building a camera from scratch, they borrowed the world's best pre-trained cameras and microphones.

DINOv2 (The Eye): Imagine a camera that has seen millions of images and learned to spot the tiniest details, like a single wrinkle forming. It doesn't just see "a face"; it sees the texture and the tiny movements.
WavLM (The Ear): Imagine a microphone that ignores the words "I am fine" but zooms in on the shaky breath or the sudden intake of air that actually reveals the person is nervous. It captures the feeling behind the voice.

B. The "Zoom-and-Pan" Detective (Hierarchical Granularity Alignment)

This is the system's way of looking at the face.

The Problem: If you zoom in too close on a mouth, you miss the head tilt. If you look at the whole head, you miss the lip twitch.
The Fix: The system uses a "HGA" module. Think of it like a detective who has a wide-angle view of the whole room (the global context) but also has a magnifying glass that snaps onto specific clues (the local muscle movements). It constantly switches between the big picture and the tiny details, making sure nothing is missed, even if the person turns their head or gets partially blocked.

C. The "Infinite Memory" Librarian (Vision-Mamba / State Space Model)

The Problem: Old systems were like people with short-term memory loss. They could only remember the last few seconds of a conversation.
The Fix: They used a new architecture called Mamba. Imagine a librarian who can read a book with 1,000 pages and remember the very first sentence while reading the last one, without getting tired or confused. This system can watch a long video and remember how the emotion started at the beginning, all while processing it very quickly (like a linear speed, not a slow, heavy one).

D. The "Audio-Visual Sync" (Audio-Guided Attention)

This is the secret sauce. The system doesn't just watch and listen separately; it lets the audio guide the eyes.

The Analogy: Imagine you are watching a silent movie, but then you hear a loud gasp. Your eyes instantly snap to the character's face to see why they gasped.
How it works: If the audio detects a sudden sigh or a pitch change, the system tells the visual part, "Hey, look right here, right now! Something important is happening." This helps the AI catch subtle facial movements that happen exactly when a sound occurs.

3. The "Fair Judge" (Asymmetric Loss)

In the wild, most of the time, people have neutral faces. Only a few seconds show a strong emotion.

The Problem: If you train a student on a test where 90% of the answers are "Neutral," the student will just guess "Neutral" every time to get a high score. They stop trying to find the rare emotions.
The Fix: The authors used a special scoring rule called Asymmetric Loss. It's like a teacher who says, "If you miss a common answer, it's okay. But if you miss a rare, difficult emotion, you get a huge penalty." This forces the AI to actually pay attention to the rare, tricky moments instead of just guessing "nothing is happening."

The Result

When they tested this system on the Aff-Wild2 dataset (a very difficult collection of real-world videos), it crushed the competition.

It won the top spot in the 10th Affective Behavior Analysis in-the-wild Competition.
It proved that by using "super-eyes," "super-ears," "infinite memory," and a "fair judge," we can finally understand human emotions in the messy, real world.

In short: They built an AI that doesn't just look at faces; it feels the moment by combining super-seeing, super-hearing, and a memory that never forgets, all while being guided by the sounds of human emotion.

1. Problem Statement

Facial Action Unit (AU) detection in "in-the-wild" environments (e.g., the Aff-Wild2 dataset) faces three primary challenges that existing methods fail to address effectively:

Spatial-Temporal Heterogeneity: Extreme variations in lighting, head pose, and occlusion make it difficult to capture subtle muscle movements.
Feature Representation Limitations: Traditional visual encoders (e.g., ResNet) lose fine-grained spatial resolution due to downsampling, while standard audio models (e.g., ASR) filter out crucial paralinguistic cues (sighs, breathing, pitch) as noise, focusing only on linguistic content.
Temporal Modeling Bottlenecks: Existing approaches rely on Temporal Convolutional Networks (TCNs) with limited receptive fields or Transformers with quadratic complexity ( $O(T^2)$ ), making them incapable of modeling ultra-long temporal dependencies without performance degradation or memory overflow.
Class Imbalance: AU datasets suffer from a severe long-tailed distribution where "neutral" frames dominate, causing models to collapse into predicting the majority class (negative samples).

2. Methodology

The authors propose a novel multimodal framework that integrates Foundation Models, Hierarchical Granularity Alignment (HGA), and State Space Models (SSM).

A. Robust Feature Extraction via Foundation Models

Instead of traditional encoders, the framework leverages pre-trained foundation models to extract high-fidelity representations:

Visual Stream: Uses DINOv2 (a self-supervised Vision Transformer). Unlike supervised models, DINOv2 preserves dense, part-level object representations and is highly sensitive to local textures and micro-expressions without task-specific fine-tuning.
Audio Stream: Uses WavLM (a masked speech pre-training model). Unlike ASR models that suppress non-verbal cues, WavLM is optimized to capture paralinguistic features (prosody, emotion, breathing) and is robust to background noise.

B. Hierarchical Granularity Alignment (HGA)

To bridge the gap between global facial context and local muscle movements, the HGA module uses facial landmarks as geometric priors:

Landmark-Guided Extraction: Facial landmarks are grouped into Regions of Interest (RoIs) (e.g., eyes, mouth). Local features are aggregated from DINOv2 patches corresponding to these RoIs.
Global Context: A global feature vector is derived via Global Average Pooling (GAP) of all patches.
Dynamic Alignment: A Multi-Head Cross-Attention mechanism allows local region features to dynamically query the global feature map. This ensures that local patches are aligned with unoccluded, complementary semantic contexts across the entire face, enhancing robustness against occlusion and pose changes.

C. Audio-Guided Selective State Space Model (AG-SSM)

To handle ultra-long video sequences efficiently, the framework replaces TCNs/Transformers with a Vision-Mamba architecture:

Linear Complexity: The SSM achieves temporal modeling with $O(N)$ linear complexity, enabling the processing of ultra-long sequences without the vanishing gradient or memory issues of Transformers.
Audio-Guided Selection: A novel Audio-Guided Selective Scan mechanism is introduced. The audio stream ( $X_a$ $X_{a}$ ) dynamically modulates the state transition parameters ( $\Delta, B, C$ $Δ, B, C$ ) of the visual stream.
- If a salient audio cue (e.g., a sigh) is detected, the model increases the temporal gate ( $\Delta$ ) to focus heavily on the current visual frame.
- If the audio is static, the model preserves historical states.
- An additional audio-driven gating mechanism emphasizes visual regions triggered by audio events.

D. Asymmetric Loss (ASL)

To address the severe class imbalance (many neutral frames vs. few active AU frames), the authors employ an Asymmetric Loss strategy:

Decoupled Focusing: It applies different focusing parameters ( $\gamma_+$ for positives, $\gamma_-$ for negatives) to independently optimize rare positive samples and abundant negative samples.
Probability Shifting: A margin parameter ( $m$ ) is used to shift the probability of negative samples. If a negative sample is predicted with high confidence (probability $> m$ ), its loss contribution is zeroed out. This forces the network to focus exclusively on ambiguous and hard-to-classify frames.

3. Key Contributions

Foundation Model Integration: First to transition AU detection from capacity-limited CNNs to foundation models (DINOv2 and WavLM), significantly improving the extraction of fine-grained visual and paralinguistic audio features.
Hierarchical Granularity Alignment: A novel module that dynamically aligns global facial semantics with local active patches using landmarks, solving the trade-off between holistic context and local detail.
Vision-Mamba for AU Detection: Introduction of the AG-SSM architecture, which solves the ultra-long temporal modeling bottleneck with linear complexity and deep audio-visual synchronization.
State-of-the-Art Performance: The framework achieved top rankings in the 10th Affective Behavior Analysis in-the-wild (ABAW) Competition AU Detection track.

4. Experimental Results

Experiments were conducted on the Aff-Wild2 dataset (12 Action Units).

Baseline Comparison: The proposed method significantly outperformed baselines using standard ViT, Whisper, and TCNs.
Ablation Study Results (Average F1 Score):
- Baseline: 36.50%
- + Foundation Models (DINOv2 + WavLM): 52.41% (+15.91% gain)
- + HGA Module: 55.18%
- + AG-SSM (Temporal Modeling): 57.82%
- + Asymmetric Loss (Full Framework): 59.45%
Key Findings: The combination of foundation models provided the largest single boost, while the AG-SSM and ASL strategies were critical for handling long-term dependencies and class imbalance, respectively.

5. Significance

This paper represents a paradigm shift in affective computing by moving away from task-specific, shallow architectures toward foundation-model-driven, state-space-based frameworks.

Robustness: It effectively handles the "in-the-wild" challenges of noise, occlusion, and pose variation.
Efficiency: By utilizing SSMs, it enables the processing of continuous, ultra-long video streams that were previously computationally prohibitive for Transformers.
Multimodal Synergy: The deep integration of paralinguistic audio cues with visual dynamics provides a more holistic understanding of human emotion than unimodal approaches.
Benchmarking: The work sets a new state-of-the-art benchmark for AU detection, demonstrating that leveraging pre-trained foundation models and advanced temporal modeling can overcome decades of limitations in spontaneous expression analysis.