💻 computer science

CF-Net: Conflict Fusion with Speaker Normalisation and Certainty Weighting for Ambivalence/Hesitancy Recognition

This paper introduces CF-Net, a deep multimodal network that leverages speaker normalization, explicit conflict fusion for cross-modal incongruence, and certainty-weighted training to achieve state-of-the-art performance in detecting ambivalence and hesitancy in unconstrained videos.

Original authors: Tung Hung Bui, Hong Hai Nguyen, Van Thong Huynh

Published 2026-07-16

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Tung Hung Bui, Hong Hai Nguyen, Van Thong Huynh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to read a room not by looking at what people are saying, but by spotting the tiny cracks where their words, their voice, and their face don't quite match up. This is the fascinating world of multimodal affect recognition, a branch of artificial intelligence that tries to understand human emotions by listening to how we speak, watching how we move, and reading what we type. Usually, when people are happy, their smile, their laugh, and their "yay!" all line up perfectly. But there's a trickier, more human feeling called ambivalence and hesitancy. This is that awkward moment when you want to say "yes" but your voice wavers, or you look confident while your words sound unsure. It's like a musical trio where the drummer is playing jazz, the guitarist is playing rock, and the singer is humming a lullaby; the "feeling" isn't in any single instrument, but in the messy, confusing clash between them. Detecting this is incredibly hard for computers because most AI is trained to spot clear, happy smiles, not these subtle, contradictory signals.

Enter CF-Net, a new digital detective built by researchers Tung Hung Bui, Hong Hai Nguyen, and Van Thong Huynh to solve this specific puzzle. They entered the 3rd Edition of the Ambivalence/Hesitancy Video Recognition Challenge (part of the ABAW 11th competition) with a system designed to stop guessing and start listening to the arguments between a person's different "senses."

The Problem: The "Identity Trap" and the "Confusion Signal"

The researchers faced a tricky situation. They had a dataset called BAH, containing video clips of people being interviewed. The goal was to spot when a speaker was feeling ambivalent. However, there were three big hurdles:

The Signal is a Mismatch: Unlike a clear "happy" face, ambivalence is defined by incongruence. If a person says something positive but their voice sounds shaky, the computer needs to notice that disagreement, not just the words or the voice alone.
The "Stranger" Problem: The test videos featured people the computer had never seen before. If the AI learned to recognize a specific person's face or voice timbre (like memorizing a friend's laugh), it would fail completely when faced with a stranger. It needed to ignore who was speaking and focus only on how they were speaking.
The "Maybe" Labels: Sometimes, even human experts couldn't agree if a clip showed ambivalence or not. The dataset included a "certainty score" for each clip, indicating how much the human annotators agreed. The AI needed to know when to trust a label and when to be careful.

The Solution: CF-Net's Three-Step Magic Trick

The team built CF-Net, a system that acts like a super-smart mediator in a three-way argument between Vision (what we see), Audio (what we hear), and Text (what is said).

Step 1: The "Ghost" Filter (Speaker Normalisation)
First, the system looks at the video and audio. But here's the trick: before analyzing the emotion, it performs a "speaker normalisation." Imagine you are trying to judge a singer's performance, but you first subtract their unique, permanent voice texture so you only hear the changes in their tone for that specific song. CF-Net does this mathematically. It calculates the average "face" and "voice" of each speaker and subtracts it out. This removes the "identity leakage"—the risk of the AI just memorizing that "Person A looks nervous"—and forces the system to focus only on the momentary hesitation or conflict.

Step 2: The "Conflict" Detector (ConflictFusion)
Next, the system takes the cleaned-up signals from the eyes, ears, and text. Instead of just smushing them together (which assumes they all agree), CF-Net uses a special module called ConflictFusion. Think of this as a referee that doesn't just listen to the players but specifically measures how far apart their opinions are. It calculates the "distance" between the text and the voice, the text and the face, and the voice and the face. If the text says "I'm great" but the voice sounds shaky, the system sees a huge gap. It turns that gap into a feature, explicitly teaching the AI that disagreement is the signal it's looking for.

Step 3: The "Honesty" Check (Certainty Weighting)
Finally, the system is trained to be humble. The researchers gave the AI a "certainty head"—an extra brain cell that tries to guess how sure the human annotators were about a clip. If the humans were unsure (low certainty), the AI is told, "Don't be too confident in your answer; this is a tricky one." This is done using a special math trick called certainty-weighted focal loss, which tells the AI to pay extra attention to the clear, obvious examples and be more gentle with the confusing ones.

The Results: A Light Touch, A Heavy Win

The team tested CF-Net on the BAH dataset. They didn't try to retrain the massive, pre-trained brains (backbones) that read the images and sounds; they kept them frozen to save time and prevent overfitting. They just trained the small "glue" parts that connect them.

The results were impressive. On the validation set (a practice test), CF-Net achieved a Macro F1 score of 0.7155. But the real magic happened on the private test set, which contained completely new speakers the AI had never seen. There, the score jumped to 0.7364 (with an Average Precision of 0.7392).

This is significant because many other systems got better on the practice test but worse on the real test (a sign they were just memorizing the practice faces). CF-Net did the opposite: it got better when faced with strangers. This proves that the "Ghost Filter" (speaker normalisation) and the "Honesty Check" (certainty weighting) actually worked, helping the AI generalize to new people rather than cheating by memorizing the old ones.

In a field where other teams spent hours fine-tuning massive models and still struggled with the "stranger" problem, CF-Net managed to beat the best previous test scores using a lightweight approach that trained in under 15 minutes on a single consumer graphics card. It showed that to understand human hesitation, you don't need to know who the person is; you just need to listen closely to where their words, voice, and face are disagreeing.

Technical Summary: CF-Net for Ambivalence/Hesitancy Recognition

Problem Definition
Detecting Ambivalence and Hesitancy (AH) in unconstrained video presents a unique challenge distinct from standard discrete emotion recognition. While emotions like happiness or disgust typically manifest as correlated signals across facial, vocal, and textual modalities, AH is characterized by cross-modal incongruence. For instance, a speaker's words may express positivity while their prosody signals hesitation, or their face remains neutral while their lexical choices convey doubt. The discriminative signal lies in the disagreement between modalities rather than in any single channel.

This challenge is further complicated by the BAH dataset constraints used in the ABAW AH challenge series:

Speaker Disjointness: Training, validation, and test sets contain entirely different speakers. Systems that inadvertently encode speaker-specific visual appearance or voice timbre (identity leakage) fail to generalize to the test set.
Data Scarcity: The training set (778 clips) is small relative to the dimensionality of large pre-trained backbones, increasing the risk of overfitting.
Annotation Uncertainty: Labels are not binary certainties but include a per-clip "certainty score" derived from annotator agreement. Low-certainty clips represent genuine perceptual ambiguity rather than labeling errors, requiring models to avoid overconfidence on borderline samples.

Methodology: CF-Net
The authors propose CF-Net, a deep multimodal network designed to address these specific challenges through frozen backbones, explicit conflict modeling, and speaker normalization.

Feature Extraction with Frozen Backbones:
- Visual: SigLIP2 (frozen) processes face crops sampled at 30 fps, producing 768-dimensional embeddings.
- Audio: HuBERT (frozen) processes 20ms audio chunks, mean-pooled to the frame level.
- Text: DistilBERT (frozen) encodes manually transcribed utterances into a single 768-dimensional vector.
- Note: All backbone weights remain frozen during training to prevent overfitting and identity memorization.
Per-Speaker Normalization:
Before temporal encoding, the visual and audio features undergo speaker normalization. For each speaker, the mean embedding across all their videos is computed and subtracted from every frame of every video belonging to that speaker. This removes the constant "identity offset" (appearance/timbre) while preserving within-speaker temporal variations (the AH signal).
Temporal Encoding:
Normalized visual and audio sequences are passed through independent bidirectional GRUs (BiGRU) with MIL (Multiple Instance Learning) attention pooling to generate 128-dimensional embeddings. Text is projected via a simple linear encoder.
ConflictFusion Module:
Instead of simple concatenation, CF-Net employs a ConflictFusion mechanism. The fused representation $f$ is constructed by concatenating the three modality embeddings ( $e_v, e_a, e_t$ ) with their pairwise absolute differences:
$f = [e_v; e_a; e_t; |e_v - e_a|; |e_v - e_t|; |e_a - e_t|]$
This explicitly encodes cross-modal incongruence, allowing the model to learn directly from the disagreement between channels.
Training Strategy & Regularization:
- Certainty-Weighted Focal Loss: The loss function weights samples based on annotator agreement scores ( $c \in [1,3]$ ). High-certainty samples receive up to 3 $\times$ more gradient weight than low-certainty ones, directing the model toward reliable signals.
- Auxiliary Certainty Head: An auxiliary regression head predicts the annotator certainty score from the fused representation. This acts as a regularizer, preventing the model from becoming overconfident on ambiguous samples where annotators disagree.
- Data Augmentation: Manifold mixup and modality dropout (15% probability) are used to further regularize the model against the small dataset size.

Key Contributions

ConflictFusion Architecture: A lightweight fusion module that explicitly represents cross-modal incongruence via pairwise absolute-difference features, tailored for the AH task where disagreement is the signal.
Speaker Normalization: A pre-encoding normalization step that removes speaker-identity offsets from visual and audio features without requiring subject-specific adaptation data, addressing the speaker-disjoint challenge.
Uncertainty-Aware Training: The integration of an auxiliary certainty-regression head and certainty-weighted focal loss to handle the inherent subjectivity and ambiguity of AH annotations.
Efficiency: A fully frozen-backbone approach that trains in under 15 minutes on a single consumer GPU while achieving competitive results.

Experimental Results
Evaluated on the BAH dataset (3rd ABAW AH Challenge):

Validation Performance: CF-Net achieved a Macro F1 of 0.7155 on the 124-clip validation set.
Test Performance: On the private challenge test set (152 clips), CF-Net achieved a Macro F1 of 0.7364 (AP = 0.7392).
Generalization: Notably, the model showed a positive validation-to-test transfer (+0.021), contrasting with other state-of-the-art methods (e.g., Team LEYA) that suffered from overfitting (validation F1 0.854 dropping to 0.714 on the test set).
Ablation: The study confirmed that removing ConflictFusion (reverting to mean fusion) dropped performance by 1.8 percentage points. The combination of the auxiliary head, focal loss, mixup, and dropout contributed an additional 3.7 points, while speaker normalization and the BiGRU encoder added a further 1.7 points cumulatively.

Significance
The paper positions CF-Net as a solution that successfully navigates the trade-off between model capacity and generalization in a data-scarce, speaker-disjoint environment. By explicitly modeling cross-modal disagreement and removing speaker identity leakage through normalization, CF-Net achieves robust performance without the overfitting risks associated with end-to-end fine-tuning of large models. The results demonstrate that a lightweight, frozen-backbone architecture, when combined with principled handling of annotation uncertainty and identity normalization, can outperform or match more complex, fine-tuned ensembles on the BAH benchmark.

The Problem: The "Identity Trap" and the "Confusion Signal"

The Solution: CF-Net's Three-Step Magic Trick

The Results: A Light Touch, A Heavy Win

Technical Summary: CF-Net for Ambivalence/Hesitancy Recognition

More like this