💻 computer science

Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction

This paper presents a two-stage multimodal framework that independently trains text, audio, vision, and motion encoders before fusing them via a lightweight regressor to predict six continuous emotion intensity dimensions, achieving third place in the Hume-ABAW10 Emotional Mimicry Intensity Challenge with a test set Pearson correlation of 0.57.

Original authors: Dinithi Dissanayake, Shaveen Silva, Ovindu Atukorala, Prasanth Sasikumar, Suranga Nanayakkara

Published 2026-05-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dinithi Dissanayake, Shaveen Silva, Ovindu Atukorala, Prasanth Sasikumar, Suranga Nanayakkara

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess how a person is feeling just by watching a video of them. But here's the twist: you aren't just guessing "happy" or "sad." You need to guess the intensity of six very specific feelings: Admiration, Amusement, Determination, Empathic Pain, Excitement, and Joy.

This paper describes a team's attempt to solve this puzzle for a competition called the Hume-ABAW10 challenge. They built a computer system that acts like a detective, piecing together clues from different sources to make its best guess.

Here is how their system works, explained simply:

1. The Problem: A Messy, Wild Dataset

The team was given thousands of video clips taken "in the wild" (not in a studio). These clips were messy:

Different Lengths: Some videos were just a second long; others were over 10,000 frames long.
Missing Clues: Sometimes the text (what the person said) was missing, but the audio and video were there.
Rare Emotions: Some feelings, like "Empathic Pain" (feeling pain for someone else), were almost never shown in the data. It was like trying to learn to recognize a rare bird when you only have pictures of sparrows.

2. The Solution: A "Two-Stage" Team of Experts

Instead of trying to teach one giant brain to do everything at once, the team built a two-stage framework. Think of it like hiring a team of specialists before bringing them together for a group meeting.

Stage 1: The Specialists Train Alone
First, they trained four separate "experts" (neural networks) on just one type of clue:

The Text Expert: Reads the transcript of what was said.
The Audio Expert: Listens to the tone of voice and sound.
The Visual Expert: Watches the facial expressions.
The Motion Expert: (Optional) Watches the subtle movements of the head and face.

Each expert learned to guess the emotions using only their specific clue. The Text and Audio experts turned out to be the strongest individual guessers. The Visual and Motion experts were weaker on their own but had unique insights.

Stage 2: The Group Meeting (Fusion)
Once the experts were trained, the team brought them together. They didn't just let them shout over each other; they used a lightweight "Fusion Regressor" (a smart manager).

The Manager's Job: It takes the notes from all the experts, combines them, and makes the final prediction.
The Safety Net: To make sure the system doesn't get lazy and rely only on the Text expert, they used a trick called "Modality Dropout." During training, they would randomly hide the Text or Audio clues, forcing the manager to learn how to use the Visual or Motion clues when the main ones were missing.

3. The "Motion" Experiment

The team added a fourth expert who looked at motion (how the face moves over time).

The Result: This motion expert didn't change the score much. It was like adding a fifth person to a committee who had a tiny bit of extra information. It helped slightly, but it wasn't the star of the show. The paper notes that while the gain was small, studying how motion helps is interesting for the future.

4. The Results: How Did They Do?

The Best Strategy: The system worked best when they fed it all four clues (Text, Audio, Vision, and Motion) and used a larger amount of training data (by mixing some of the test data into the training set).
The Score: In the competition, their system achieved an average correlation of 0.57 on the final test. This means their guesses were moderately aligned with the human judges' scores.
Ranking: They placed 3rd out of all the teams in the challenge.

5. Key Takeaways

Words and Voice Win: If you had to pick just one clue, reading the transcript or listening to the voice was the most powerful way to guess the emotion intensity.
Faces and Movement Help: Seeing the face and movement didn't work as well alone, but they added a little bit of extra value when combined with words and voice.
Simple is Good: Their method was relatively simple compared to the winners. They didn't use complex, heavy machinery; they just trained specialists well and then let them work together.

In a nutshell: The team built a system that first trains individual experts on text, sound, and video separately, then combines their opinions to guess how intense a person's emotions are. They came in third place, proving that a simple, staged approach works well for this tricky task.

Technical Summary: Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction

Problem Definition

The paper addresses the Emotional Mimicry Intensity (EMI) prediction task, a challenge within the Hume-ABAW10 competition. Unlike categorical emotion recognition, EMI requires predicting continuous intensity values across six specific affective dimensions: Admiration, Amusement, Determination, Empathic Pain, Excitement, and Joy. The task is complicated by "in-the-wild" data characteristics, including strong temporal variations, partial modality missingness (specifically text), and highly imbalanced target distributions (e.g., Empathic Pain is predominantly zero-valued).

Methodology

The authors propose a two-stage multimodal framework designed to handle data sparsity and modality imbalance while ensuring robust feature learning.

1. Data Characteristics and Preprocessing

Dataset: 12,660 samples (8,072 training, 4,588 validation) with highly variable sequence lengths (median ~65 frames, but with long-tail outliers up to 12,500 frames).
Modality Handling:
- Text: Derived from transcripts using a pretrained GTE-base model. Missing text embeddings (approx. 6% of samples) are handled by substituting zero vectors.
- Audio: Extracted using a customized Wav2Vec 2.0 (16kHz sampling).
- Visual: Extracted from cropped face frames using DINOv2.
- Motion: An optional branch using OpenFace-derived facial action units (AU) and head-pose features, summarized as a 23-dimensional temporal sequence.
Exploratory Findings: The dataset exhibits long-tailed sequence distributions, significant label sparsity (especially for Empathic Pain), and distribution shifts between training and validation sets.

2. Two-Stage Training Strategy

Rather than training a single end-to-end model, the framework employs a staged approach:

Stage 1: Unimodal Pretraining: Modality-specific encoders (Text, Audio, Vision, and optional Motion) are trained independently to convergence.
- Encoders: Text uses an MLP; Audio, Vision, and Motion use attention-based temporal pooling to handle variable-length sequences.
- Objective: A combined loss function of Concordance Correlation Coefficient (CCC) and Mean Squared Error (MSE) ( $L = 0.7 L_{CCC} + 0.3 L_{MSE}$ ) is used to address label imbalance and ensure both magnitude accuracy and trend agreement.
Stage 2: Fusion and Fine-tuning:
- The unimodal prediction heads are removed, retaining the learned penultimate representations.
- A lightweight Fusion Regressor (MLP) concatenates the modality embeddings to predict the six intensity scores.
- Adaptation: During fusion, the fusion head is trained at the base learning rate, while the pretrained encoders are fine-tuned with a significantly reduced learning rate (5% of the base rate) to allow cross-modal adaptation without destabilizing unimodal representations.
- Robustness: Modality dropout (probability $p=0.3$ ) is applied during fusion training to prevent over-reliance on any single channel and simulate missing data scenarios.

3. Experimental Configurations

The authors submitted five systems varying by:

Modality Composition: Text-Audio-Vision (T+A+V) vs. Text-Audio-Vision-Motion (T+A+V+M).
Data Split: Standard 2:1 train/valid split vs. an expanded 4:1 split (incorporating original validation samples into training to increase data volume).
Encoder Strategy: Joint fine-tuning vs. frozen encoders.

Key Results

Unimodal Performance: Text and Audio provided the strongest individual signals.
- Text: Achieved the highest average Pearson correlation ( $\bar{r} = 0.395$ ), particularly strong for Admiration and Empathic Pain.
- Audio: Performed second best ( $\bar{r} = 0.350$ ) with balanced performance across dimensions.
- Vision & Motion: Performed significantly lower in isolation ( $\bar{r} \approx 0.15$ and $0.13$, respectively) but showed complementary value for specific dimensions like Joy, Amusement, and Excitement.
Fusion Performance:
- The Text-Audio-Vision-Motion (T+A+V+M) model on the expanded 4:1 split achieved the best validation performance with an average Pearson correlation of 0.4722.
- Adding the motion branch provided only marginal gains (e.g., 0.4717 to 0.4722 on validation), suggesting it acts as an auxiliary enhancement rather than a dominant signal.
- The best test set performance was achieved by the Text-Audio-Vision (T+A+V) model on the 4:1 split, achieving an average Pearson correlation of 0.57.
Ranking: The team placed third in the Hume-ABAW10 EMI Challenge.

Contributions and Significance

The paper claims three primary contributions:

A Practical Staged Training Strategy: A reproducible pipeline that stabilizes multimodal learning on sparse and imbalanced affective targets by decoupling unimodal pretraining from cross-modal fusion.
Empirical Modality Analysis: A detailed breakdown of how different modalities contribute to specific emotion axes, revealing that semantic (text) and acoustic (audio) cues are the primary drivers, while visual and motion cues offer selective, complementary information.
Motion Cues Exploration: A submission highlighting the role of motion in multimodal fusion. While the gains were slight, the authors posit that motion remains a promising direction for future exploration in capturing temporal facial dynamics.

The authors emphasize that their framework offers a simple and reproducible baseline for EMI prediction. They acknowledge that while their method is structurally simpler than top-ranked competitors (which utilized complex cross-attention mechanisms and multi-objective optimization), their staged approach effectively addresses the specific challenges of data sparsity and modality missingness inherent in the dataset.