💻 computer science

Foundation Model Embeddings Meet Blended Emotions: A Multimodal Fusion Approach for the BLEMORE Challenge

This paper presents a 12-encoder multimodal fusion system for the BLEMORE Challenge that achieves 6th place by combining specialized face, audio, and body-language models with the novel application of Gemini Embedding 2.0, demonstrating that frozen prosody layers and personalized salience thresholds are critical for recognizing blended emotions.

Original authors: Masoumeh Chapariniya, Aref Farhadipour, Sarah Ebling, Volker Dellwo, Teodora Vukovic

Published 2026-03-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Masoumeh Chapariniya, Aref Farhadipour, Sarah Ebling, Volker Dellwo, Teodora Vukovic

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a crowded party. Someone walks up to you, and they aren't just feeling one thing. They might be 70% excited but 30% nervous. Or maybe they are 50% angry and 50% sad. This mix of feelings is called a "blended emotion."

The BLEMORE Challenge is like a high-stakes game where computers try to guess these mixed feelings just by watching short video clips of people. The tricky part isn't just guessing what emotions are there, but also figuring out how strong each one is (the "salience").

Here is how the team from the University of Zurich built a "super-detective" system to win 6th place in this challenge, explained in simple terms.

1. The Detective Squad (The Multimodal Ensemble)

Instead of hiring just one detective, the team built a squad of six different experts. Each expert looks at the video from a different angle, and then they all sit around a table to vote on the final answer.

The Face Expert (S4D-ViTMoE): This detective only looks at the person's face. They are trained to spot tiny muscle twitches that show happiness or fear.
The Body Language Experts (TimeSformer & VideoMAE): Sometimes a face is a poker face, but the body screams the truth. These experts watch the person's shoulders, hands, and posture. If someone is clenching their fists, they know it's anger, even if the face is calm.
The Sound Expert (Wav2Vec2): This one listens to the voice. But here's the trick: in this challenge, people aren't speaking words; they are making sounds like gasps, laughs, or growls. The team realized that the "middle layers" of this AI (like the middle chapters of a book) are best at understanding the rhythm and pitch of these sounds, while the "top layers" are too busy trying to understand words. So, they ignored the word-hunters and only used the rhythm-hunters.
The "Super-Brain" (Gemini Embedding 2.0): This is the star of the show. It's a massive, general-purpose AI (like a super-smart robot that has seen the whole internet). The team used it for the first time in this field. Amazingly, this robot only needed to see the first 2 seconds of a video to guess the emotions correctly. It's like a detective who can solve a crime just by glancing at the front door, without needing to search the whole house.
The Backup Team: They also added a few other pre-trained tools provided by the challenge organizers to make sure they didn't miss anything.

2. The Voting System (Late Fusion)

Once all six experts give their opinion (e.g., "I think it's 60% fear and 40% joy"), the team doesn't just pick the loudest voice. They use a weighted vote.

Think of it like a jury. Some jurors are more experienced than others. The team found that the "Super-Brain" (Gemini) and the "Face Expert" were the most reliable, so they got the most votes. The others got fewer votes, but their input still helped correct mistakes.

3. The "Threshold" Problem (The Shaky Ruler)

This is the most interesting part of their discovery. To turn the AI's fuzzy guesses (like "70% fear") into a final answer, the system uses a "ruler" called a threshold.

The Problem: The team found that this ruler was unstable. For one actor, the ruler needed to be set very low to catch a subtle emotion. For another actor, the ruler needed to be set very high.
The Analogy: Imagine trying to guess how loud a person is whispering. If you are talking to a quiet person, a whisper sounds loud. If you are talking to a loud person, the same whisper sounds quiet. The "ruler" changes depending on who is speaking.
The Result: This "personal style" of each actor was the biggest reason the computer struggled. Even the best AI couldn't perfectly guess the mix because every human expresses emotions differently.

4. The Big Wins

The team learned three major lessons:

Don't reinvent the wheel: For the sound part, they didn't need to teach the AI everything from scratch. Just picking the right "middle layers" of an existing AI worked better than training a new one.
Big brains are fast: The giant "Super-Brain" (Gemini) was so smart it could guess emotions correctly from just a 2-second clip, proving that huge AI models are ready for this kind of work.
Humans are unique: The hardest part of the challenge wasn't the computer's intelligence; it was that humans are all different. A computer can learn the rules, but it can't easily predict how you specifically will express your feelings.

The Final Score

By combining all these experts and voting together, the team achieved a score of 0.279, beating the previous best attempts. They proved that if you want to understand mixed emotions, you need to look at the face, the body, the sound, and use a super-smart AI to tie it all together.

In short: They built a team of specialists who combined their unique skills to read the complex, messy, and very human world of blended emotions.

1. Problem Definition

The paper addresses the BLEMORE Challenge (Blended Emotion Recognition with Relative Salience), which focuses on recognizing complex human emotional states where multiple emotions co-occur with varying degrees of prominence.

Core Task: The system must solve two coupled problems:
1. Presence Recognition: Identifying which emotions are present in a video clip.
2. Salience Recognition: Determining the relative prominence (e.g., 70/30 or 50/50 split) of the co-occurring emotions.
Dataset Characteristics: The BLEMORE dataset contains 3,050 video clips from 58 actors. A critical feature is that the audio consists of non-linguistic vocalizations (laughs, cries, gasps) rather than spoken words, making traditional phonetic analysis irrelevant.
Evaluation Metric: Performance is measured by a composite Score ( $0.5 \times ACC_P + 0.5 \times ACC_S$ ), where $ACC_P$ is presence accuracy and $ACC_S$ is salience accuracy. Predictions are discretized using two thresholds: a presence threshold ( $\alpha$ ) and a salience threshold ( $\beta$ ).

2. Methodology

The authors propose a multimodal late-fusion ensemble consisting of six distinct encoder families. The pipeline processes video clips (3–8 seconds) and audio waveforms to generate probability distributions over six emotion classes, which are then fused and post-processed.

A. Visual Encoders

S4D-ViTMoE (Face):
- Architecture: Adapts the S4D (State-Space Model) Vision Transformer with Mixture-of-Experts (MoE) adapters.
- Adaptation: Pretrained on talking-head videos (VoxCeleb2) and static faces (AffectNet). The authors use face-aligned preprocessing (MTCNN crops) to match the pretraining domain.
- Training: Replaces standard cross-entropy with KL Divergence loss on soft labels to preserve the exact blend ratio (e.g., [0.7, 0, 0.3...]). Only the MoE adapter layers and head are finetuned; the backbone is frozen.
- Sampling: Uses a temporal sampling strategy (16 frames from a 64-frame window) and multi-clip testing to reduce temporal sensitivity.
Body Language Encoders (TimeSformer & VideoMAE):
- Purpose: To capture gestural and postural cues that faces alone miss, especially for blended emotions.
- Input: Full-body frames (no face cropping).
- Training: Finetuned on Kinetics-400/600 checkpoints using soft-label KL training.

B. Audio Encoder

Layer-Selective Wav2Vec2:
- Insight: Since BLEMORE audio is non-verbal, the upper layers of Wav2Vec2 (optimized for phonetics) are irrelevant.
- Strategy: Extracts hidden states from frozen layers 6–12 (which encode prosody and paralinguistics), averages them, and feeds them into a lightweight MLP classifier.
- Result: This approach significantly outperforms end-to-end finetuning, avoiding overfitting on the small dataset.

C. Foundation Model Embeddings (Novel Contribution)

Gemini Embedding 2.0:
- Innovation: First application of Large Multimodal Model (LMM) embeddings for blended emotion recognition.
- Constraint: Limited to the first 2 seconds of video input due to API constraints.
- Performance: Surprisingly competitive, achieving high presence accuracy despite the short temporal window.

D. Fusion and Post-Processing

Late Fusion: Probability vectors from all encoders are combined via weighted averaging. Weights are optimized via constrained grid search on validation folds.
Post-Processing: Converts continuous probabilities to discrete predictions using:
1. Top-2 masking.
2. Presence threshold ( $\alpha$ ).
3. Salience threshold ( $\beta$ ): If the difference between top two probabilities is $\le \beta$ , the output is 50/50; otherwise, it is 70/30.

3. Key Contributions

S4D-ViTMoE Adaptation: A complete pipeline adapting a state-of-the-art video transformer for blended emotions using soft-label KL training and domain-matched preprocessing.
Layer-Selective Audio Features: Demonstrated that freezing Wav2Vec2 and selecting specific prosody-sensitive layers (6–12) outperforms end-to-end finetuning by 22% on non-verbal emotion tasks.
LMM Embeddings for Emotion: Introduced Gemini Embedding 2.0 as a feature extractor, showing that foundation models can capture rich emotional semantics from just 2 seconds of video, outperforming domain-specific encoders trained on full clips.
Threshold Sensitivity Analysis: Quantified the instability of the salience threshold ( $\beta$ ), showing it varies from 0.05 to 0.43 across different validation folds. This reveals that personalized expression styles are the primary bottleneck, not model capacity.

4. Experimental Results

Test Set Performance: The 12-encoder ensemble achieved a Score of 0.279 ( $ACC_P = 0.391$ , $ACC_S = 0.168$ ), placing 6th in the challenge.
Comparison to Baselines:
- Outperformed the best baseline (VideoMAEv2 + HuBERT, Score 0.223) by a significant margin.
- Adding Gemini Embeddings improved the score from 0.262 to 0.279 (+6.5%).
Encoder Weights: In the final ensemble, task-adapted encoders (S4D, Wav2Vec2, Body models, Gemini) received 62% of the total weight, while pre-extracted baseline features received 38%. Gemini received the highest individual weight (0.192) in the 12-encoder setup.
Individual Encoder Highlights:
- S4D-ViTMoE: Best single encoder for presence accuracy (0.340).
- Gemini 2.0: Achieved $ACC_P = 0.317$ using only 2 seconds of video.
- Frozen Wav2Vec2: Score 0.207 vs. 0.161 for end-to-end finetuning.

5. Significance and Future Directions

Methodological Insight: The paper proves that for small-scale, non-verbal emotion datasets, selecting the right pretrained layers is more effective than architectural adaptation or end-to-end training.
Foundation Models: It establishes a new direction for affective computing, showing that LMMs can serve as powerful, general-purpose feature extractors for fine-grained emotion tasks, even with limited temporal input.
Evaluation Challenge: The analysis of the salience threshold ( $\beta$ ) highlights a fundamental limitation in current evaluation protocols. The high variability of $\beta$ across actors suggests that personalized expression styles are the main hurdle. Future work should explore actor-adaptive normalization or continuous salience regression to eliminate threshold dependency.

Conclusion: The system successfully combines specialized domain adaptations with general-purpose foundation models to achieve state-of-the-art results in blended emotion recognition, while providing critical insights into the limitations of current evaluation metrics regarding actor variability.