Team LEYA in 10th ABAW Competition: Multimodal Ambivalence/Hesitancy Recognition Approach

Team LEYA's multimodal approach for the 10th ABAW Competition integrates scene, face, audio, and text modalities using advanced encoders and prototype-augmented fusion strategies to achieve a 71.43% MF1 score, significantly outperforming unimodal baselines in recognizing ambivalence and hesitancy in unconstrained videos.

The Gist

Imagine you are trying to guess if someone is hesitating or feeling conflicted about a decision just by watching a video of them. Maybe they are talking about whether to quit their job or change their diet.

This is the challenge Team LEYA tackled in the 10th ABAW competition. Their goal was to build a computer program that can spot these subtle "I'm not sure" moments in videos.

Here is how they did it, explained simply:

The Problem: The "Mixed Signal" Puzzle

Hesitation is tricky. It's not like a big smile (happiness) or a loud scream (anger). It's subtle.

• The Analogy: Imagine a person saying, "I love this new job!" with a huge smile, but their voice is shaking, and they are nervously tapping their foot.
• The Conflict: Their words say "Yes," but their body and voice say "Maybe not." This mismatch is called ambivalence. To catch it, a computer can't just listen to the words; it has to watch the face, the background, and the tone of voice all at once.

The Solution: The "Four-Sense Detective Team"

Team LEYA didn't rely on just one sense. They built a team of four specialized AI detectives, each looking at a different part of the video:

1. The Scene Detective (The Background):

   • What it does: It looks at the whole room, not just the person. Is the room chaotic? Is the lighting dim?
   • The Metaphor: Think of this as the "vibe check." Sometimes the environment tells you if someone is stressed or unsure, even if they are trying to hide it. They used a high-tech model called VideoMAE to understand the flow of the scene.

2. The Face Detective (The Expression):

   • What it does: It zooms in on the person's face to catch micro-expressions.
   • The Metaphor: This is like a forensic artist looking for a tiny twitch of the eyebrow or a fleeting frown that lasts only a split second. They took thousands of snapshots of the face and averaged them out to find the "emotional fingerprint."

3. The Voice Detective (The Tone):

   • What it does: It analyzes the audio, looking for pauses, shaky tones, or changes in pitch.
   • The Metaphor: This is the "lie detector" for sound. It listens for the hesitation in a voice that says "I'm fine" but sounds like they are about to cry. They used a model called Mamba to understand the rhythm of the speech.

4. The Text Detective (The Words):

   • What it does: It reads the transcript of what the person is saying.
   • The Metaphor: This is the "logic checker." It looks for words like "maybe," "I guess," or "on the one hand." In this specific challenge, the words turned out to be the strongest single clue.

The "Brain": Fusing the Clues

Having four detectives is great, but they need a Chief Investigator to put the clues together.

• The Fusion Model: The team built a special AI "brain" (a Transformer) that takes the reports from all four detectives.
• The "Prototype" Trick: To make the Chief Investigator even smarter, they added a special tool called Prototype-Augmentation.
  • The Analogy: Imagine the Chief Investigator has a mental "file cabinet" with perfect examples of "Total Certainty" and "Total Hesitation." When a new video comes in, the system compares the clues against these perfect examples to see which one it matches better. This helps the AI make a more confident guess.

The Results: Teamwork Wins

The team tested their system on a dataset of real people answering questions.

• Solo Acts: When they let just the "Text Detective" work alone, it got about 70% of the answers right. The "Face Detective" alone was much worse (around 62%).
• The Power of the Team: When they combined all four detectives, the score jumped to 83% on the practice tests.
• The Grand Finale: For the final competition, they didn't just use one Chief Investigator. They created an Ensemble—a council of five different investigators who all voted on the answer. This "wisdom of the crowd" approach helped them reach 71.43% accuracy on the final, unseen test data.

The Big Takeaway

The paper proves that to understand human hesitation, you can't just look at one thing. You need to listen to the words, watch the face, hear the tone, and check the surroundings.

The Lesson: Just like a human detective needs to look at all the evidence to solve a mystery, an AI needs to combine all the senses to understand the complex, messy reality of human emotion. By combining these different "senses" and using a smart voting system, Team LEYA built a much better "hesitation detector" than anyone else in the competition.

Technical Summary

1. Problem Definition

The paper addresses the Ambivalence/Hesitancy (A/H) Recognition challenge within the 10th Workshop and Competition on Affective & Behavior Analysis in-the-Wild (ABAW).

• Task: Binary classification at the video level to determine if a video contains signs of ambivalence or hesitancy.
• Challenge: A/H is a subtle, context-dependent behavioral state often characterized by inconsistencies across modalities (e.g., a person saying they are willing while their tone or facial expression suggests resistance). Unlike basic emotions, A/H requires detecting these cross-modal discrepancies.
• Dataset: The BAH corpus, containing 1,427 videos from 300 participants in a digital behavior change scenario. The dataset includes synchronized video, audio, and text transcripts, with expert annotations for A/H segments.

2. Methodology

The proposed approach, Team LEYA, employs a four-stage multimodal pipeline integrating Scene, Face, Audio, and Text modalities.

A. Unimodal Feature Extraction

Four dedicated models extract representations for each modality:

1. Scene-based Visual Model:
   • Architecture: VideoMAE (based on ViT) pre-trained on Kinetics-400.
   • Input: 16 uniformly sampled frames ($224 \times 224$).
   • Output: A compact scene embedding ($h_s$) derived via global average pooling of spatio-temporal tokens.
2. Face-based Visual Model:
   • Architecture: EmotionEfficientNetB0 (fine-tuned on AffectNet+).
   • Process: Detects faces (using YOLO), crops, and extracts emotional embeddings per frame.
   • Aggregation: Uses Statistical Pooling (concatenating mean $\mu$ and standard deviation $\sigma$ of frame embeddings) followed by an MLP.
3. Acoustic Model:
   • Architecture: EmotionWav2Vec2.0 (fine-tuned on MSP-Podcast) for feature extraction.
   • Temporal Encoding: A Mamba encoder (state-space model) processes the sequence of acoustic embeddings to capture temporal dependencies.
   • Aggregation: Mean pooling over the temporal dimension to produce a compact acoustic embedding.
4. Linguistic Model:
   • Architecture: Fine-tuned EmotionDistilRoBERTa.
   • Process: Processes speech transcripts. The output representation is passed through an MLP classification head.
   • Note: Other text models (TF-IDF, CatBoost, other Transformers) were evaluated, but EmotionDistilRoBERTa yielded the best unimodal performance.

B. Multimodal Fusion Strategy

The unimodal embeddings are fused using a Transformer-based fusion module:

• Projection: Each modality embedding is projected into a shared latent space ($d=128$) using a linear layer, LayerNorm, GELU, and dropout.
• Token Stacking: Projected tokens are stacked into a matrix. A learnable modality embedding is added to the sequence to handle missing modalities via masking.
• Fusion: A stack of 6 Transformer encoder layers with 4 attention heads processes the tokens.
• Output: Masked mean pooling over the output tokens generates the final fused representation ($z_{fused}$).

C. Prototype-Augmented Classification

To enhance robustness, the authors introduce a prototype-based classification objective:

• Mechanism: Learnable class-specific prototypes ($P_c$) are compared with the fused representation using a similarity score (log-sum-exp of dot products).
• Loss Function: The total loss combines the main classification loss ($L_{cls}$), an auxiliary prototype loss ($L_{proto}$), and a diversity regularization term ($L_{div}$).
• Ensemble: The final system aggregates predictions from five independently trained prototype-augmented models (using different random seeds) to improve stability and generalization.

3. Key Contributions

1. Incorporation of Scene Context: Unlike previous works focusing primarily on face, audio, and text, this approach explicitly models scene dynamics using VideoMAE, capturing environmental context relevant to behavioral uncertainty.
2. Advanced Temporal Modeling: The use of Mamba for acoustic temporal encoding and VideoMAE for scene dynamics provides a modern alternative to standard LSTM/GRU architectures.
3. Prototype-Augmented Fusion: The introduction of learnable prototypes within the fusion module acts as a regularizer, helping the model handle uncertainty and cross-modal inconsistencies characteristic of A/H.
4. Comprehensive Ablation: The study rigorously evaluates unimodal baselines, various fusion combinations, and the impact of the prototype head, demonstrating that text is the strongest single cue but multimodal fusion is essential for peak performance.

4. Experimental Results

Experiments were conducted on the BAH corpus using Macro F1-score (MF1) as the primary metric.

• Unimodal Performance:

  • Text (EmotionDistilRoBERTa): 70.02% (Best unimodal).
  • Audio (Wav2Vec2.0 + Mamba): 69.03%.
  • Face: 62.67%.
  • Scene: 61.96%.
  • Observation: Text is the dominant modality, but other modalities provide complementary information.

• Multimodal Fusion Performance:

  • Best Single Fusion Model (Prototype-Augmented): 83.25% average MF1 (Development/Validation).
  • Best Final Test Performance: 71.43% MF1.
  • Strategy: The highest test score was achieved by an ensemble of five prototype-augmented fusion models.
  • Comparison: The ensemble significantly outperformed single fusion models (71.43% vs. 68.32% for the non-prototype single model).

• Ablation Insights:

  • Scene + Text: The strongest two-modality pair (80.39% avg), suggesting scene context strongly complements linguistic cues.
  • Four-Modality Fusion: Combining all four modalities yielded the most effective overall solution, confirming that no single modality is sufficient for robust A/H detection.

5. Significance

This work demonstrates that ambivalence and hesitancy are best detected through a holistic, multimodal approach that respects the unique characteristics of each signal source.

• Robustness: The ensemble of prototype-augmented models proved crucial for generalizing to the private test set, highlighting the importance of model stability in unconstrained video analysis.
• Complementarity: The results validate that while text is the strongest indicator, integrating scene and facial dynamics (even when they are individually weaker) significantly boosts performance by capturing the inconsistencies that define A/H.
• State-of-the-Art: The approach set a new benchmark for the 10th ABAW competition, achieving 71.43% MF1, and provides a reproducible framework (code available on GitHub) for future research in subtle behavioral analysis.