💻 computer science

Multimodal Ambivalence and Hesitancy Recognition via Cross-Attention and Gated Fusion

This paper presents a multimodal framework for recognizing ambivalence and hesitancy in video, which fuses text, audio, and visual features via bidirectional cross-attention and a Gated Multimodal Unit to achieve a Macro F1 score of 0.7394, significantly outperforming both unimodal baselines and zero-shot models in the ABAW11 challenge at ECCV 2026.

Original authors: Oussama Berhili, Yassine Ouzar, Larbi Boubchir

Published 2026-07-20

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Oussama Berhili, Yassine Ouzar, Larbi Boubchir

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess what a friend is feeling just by watching them. Sometimes, it's easy: a big smile means happy, a frown means sad. But what about those tricky moments when someone is torn? Maybe they are saying "I'm fine" with a voice that sounds shaky, while their face looks like they are about to cry. This confusing mix of feelings is called ambivalence or hesitancy. It's that internal tug-of-war between wanting to do something and being scared to do it. Scientists who study computers and human behavior (a field called affective computing) are trying to teach machines to spot these subtle, mixed-up signals. They know that looking at just one clue—like only listening to the voice or only watching the face—is often not enough. To really understand a person, a computer needs to act like a super-smart detective, piecing together words, tone, and expressions all at once to solve the mystery of what someone is truly feeling.

In this paper, a team of researchers named Oussama, Yassine, and Larbi decided to build a super-detective computer to solve exactly this puzzle. They entered a competition called ABAW11, where the goal was to spot ambivalence and hesitancy in video recordings. They started by testing three different "senses" on their own: a brain that reads text transcripts, a brain that listens to audio, and a brain that watches facial videos. They found that the text reader was the best solo detective, getting about 66.6% of the answers right (measured by a score called Macro F1). The audio listener was close behind at 62.8%, and the video watcher was a bit weaker at 57.3%. Interestingly, they tried a famous "zero-shot" AI (one that wasn't trained on this specific data) and it only got 28.3% right, proving that you really need to train your AI specifically for this tricky job.

But the real magic happened when they stopped letting the detectives work alone and made them work as a team. The researchers built a special system where the text, audio, and video brains could talk to each other. They used a technique called cross-attention, which is like giving each detective a walkie-talkie to ask the others, "Hey, I see a shaky voice here, does your face see a nervous expression?" and "I see a hesitant word, does your audio hear a pause?" This allowed them to spot contradictions, like when someone says "yes" but sounds unsure. They also added a Gated Multimodal Unit, which acts like a smart manager. If one detective is looking at a blurry face or hearing bad audio, the manager can say, "Ignore that noisy signal for a second, let's focus on the clear text instead."

After training this team with a smart search tool called Optuna to find the perfect settings, the results were impressive. The team of three working together scored a 73.9% on their test, which is a 11.0% improvement over the best single detective. This suggests that by letting the different senses cross-check each other, the computer can catch the subtle, conflicting clues that make up ambivalence much better than any single sense could alone. The researchers are confident that this method of explicit teamwork between text, sound, and video is the key to unlocking these difficult human emotions, and they have made their code public so others can try it out too.

Technical Summary: Multimodal Ambivalence and Hesitancy Recognition via Cross-Attention and Gated Fusion

Problem Definition
This work addresses the challenge of automatically detecting Ambivalence and Hesitancy (A/H) in video recordings, a task central to the ABAW11 challenge at ECCV 2026. A/H represents a subtle affective state where an individual experiences conflicting emotions or motivations, often leading to delays in behavioral change. The challenge lies in the fact that A/H manifests through complex, sometimes conflicting cues across multiple modalities: lexical hedging in transcripts, prosodic variations in audio, and micro-expressions in facial video. The authors utilize the BAH dataset, which contains 1,427 video recordings with binary labels indicating the presence or absence of A/H, to develop a solution that moves beyond simple unimodal analysis or naive fusion.

Methodology
The proposed approach is a structured pipeline involving feature extraction, unimodal baseline establishment, and a novel multimodal fusion architecture.

Feature Extraction: To preserve fine-grained temporal dynamics, the authors avoid global pooling during extraction. Instead, they extract full-sequence token representations using three state-of-the-art pretrained encoders:
- Text: F2LLM-v2-0.6B (based on Qwen3-0.6B) processes transcripts, retaining the full last_hidden_state sequence ( $N_t \times 1024$ ).
- Audio: WavLM-Large processes 16 kHz mono audio, extracting the full temporal sequence ( $N_a \times 1024$ ) to capture prosodic nuances.
- Visual: VideoMAE V2 (Base) processes aligned facial frames (256x256). Videos are split into 16-frame chunks, and patch-level token sequences are concatenated to preserve spatio-temporal context ( $N_v \times 768$ ).
Unimodal Baselines: Before fusion, the authors establish robust baselines by training classical classifiers (MLP, Random Forest, and GBDT) on each modality independently. These models are optimized via Optuna (100 trials per classifier) using Macro F1 as the metric.
Multimodal Fusion Architecture: The core contribution is a fusion model combining bidirectional cross-attention and a Gated Multimodal Unit (GMU):
- Projection & Positional Encoding: Variable-length sequences from all modalities are linearly projected to a shared embedding dimension ( $d_{model}$ ) and augmented with sinusoidal positional encodings.
- Bidirectional Cross-Attention: A MultimodalCrossAttention module implements six directional attention stacks (e.g., text attending to audio, audio to video, etc.). This allows each modality to query and attend to the representations of the others, resolving ambiguities by leveraging complementary cues.
- Pooling: Contextualized token sequences are reduced to single vectors using a configurable pooling strategy (mean, max, or learned attention).
- Gated Fusion (GMU): The pooled vectors are concatenated and processed by a GMU. This unit computes a context vector to generate a sigmoid gate for each modality, dynamically scaling the contribution of each stream. This mechanism allows the model to suppress noisy or uninformative modalities (e.g., occluded faces) for specific instances.
- Classification: The final fused representation passes through an MLP to output a binary prediction.

Key Contributions

Comprehensive Unimodal Benchmarks: The paper establishes strong unimodal baselines using specific configurations of F2LLM, WavLM, and VideoMAE, demonstrating that domain-specific training significantly outperforms zero-shot baselines.
Systematic Classifier Evaluation: A comparative analysis of MLP, Random Forest, and GBDT classifiers across modalities, optimized via Optuna.
Novel Fusion Architecture: The integration of bidirectional cross-attention with a Gated Multimodal Unit, specifically tuned via a 50-trial Optuna search, to model both intra- and inter-modal interactions.
Reproducible Pipeline: A modular, open-source framework that preserves variable-length token sequences throughout the pipeline, facilitating future extensions to cross-attention or multimodal LLMs.

Results

Unimodal Performance: Text features proved the most discriminative in isolation, with a Random Forest classifier achieving a Macro F1 of 0.6659 on the test set. This substantially outperformed the zero-shot Video-LLaVA baseline (Macro F1: 0.2827). Audio and visual modalities followed with Macro F1 scores of 0.6275 and 0.5727, respectively.
Multimodal Performance: The proposed Cross-Attention + GMU model achieved a Macro F1 of 0.7394 on the validation set. This represents an 11.0% relative improvement over the best unimodal baseline and a significant gain over the zero-shot baseline.
Hyperparameter Optimization: The best configuration utilized a shared embedding dimension of 256, a single attention head, two cross-attention layers per modality pair, and mean pooling.

Significance and Claims
The authors claim that their results confirm the hypothesis that explicit cross-modal interaction is essential for detecting A/H. The performance gain over unimodal models suggests that A/H involves complementary cues that are not captured by any single modality in isolation. Furthermore, the error analysis indicates that while unimodal models (particularly text) are sensitive to markers, they may over-trigger on neutral features; the multimodal architecture helps refine the decision boundary by leveraging orthogonal cues from audio and visual streams.

The paper positions its approach as a principled step forward in affective computing, demonstrating that modeling interactions within and across modalities yields superior results compared to naive late fusion or reliance on single modalities. While the performance (0.7394) approaches the results of the ABAW10-winning BROTHER system, the authors note that direct comparison is cautious due to differences in dataset versions and challenge editions. The work concludes that future improvements may lie in richer temporal modeling, personalization, and parameter-efficient fine-tuning of multimodal LLMs.

More like this