⚡ electrical engineering

Simple Features and Honest Calibration for Ambivalence and Hesitancy Recognition in Video

This paper presents a system for recognizing ambivalence and hesitancy in video interviews that leverages ASR-erased time features and affect-specialized multimodal fusion, demonstrating that simple AP-weighted ensembling with honest calibration outperforms complex architectures and overfitted validation strategies on the ABAW 2026 BAH Challenge.

Original authors: Vikas Kumar, Aditya Mishra, Haroon R. Lone

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vikas Kumar, Aditya Mishra, Haroon R. Lone

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: Is this person feeling torn between two choices, or are they just hesitating? The clues are hidden in a short video of someone answering interview questions. This paper is the detective's report on how they cracked the case using a new set of tools, and more importantly, what didn't work.

The Big Mystery: Finding the "Wobble"

The goal was to spot Ambivalence and Hesitancy (A/H)—that internal "I'm not sure" feeling where a person is pulled in two directions. The team tested their detective skills on a dataset called BAH, which contains 1,427 videos from 300 different people. They had to guess if the person in the video was showing signs of being unsure.

The "Magic" Clue: The Ghost in the Transcript

The biggest surprise? The team found a "ghost" signal that nobody else was looking at.

Imagine a speech-to-text robot (an ASR) listening to the video. Its job is to write down the words. To make the text clean, the robot deletes "um," "uh," and long pauses. But here's the trick: it can't delete the time those sounds took.

The team realized that even though the words "um" are gone, the empty space where they used to be is still there in the timestamps. They built a new tool called "ASR-erased time." It measures the gaps between the remaining words.

Why it's cool: This gap-measuring tool turned out to be the strongest non-verbal clue they found. It scored an AP of 0.718 (a measure of how good the clue is).
The independence: It was so unique that it barely agreed with any other clue (only 0.11–0.36 correlation). It was like finding a fingerprint that no other detective had ever seen before.

The "Text" is the Star, The Face is a Sidekick

The team tested three main types of clues: Text (what they said), Audio (how they sounded), and Video (what their face looked like).

Text is King: The words people used were by far the best clue. When they used a special text model trained on emotions, it scored an AP of 0.811. Even a simple list of "hesitation markers" (like counting how many times someone said "but" or "maybe") was almost as good (AP 0.800).
Audio is the Runner-Up: If the audio model was trained specifically to detect emotions (not just speech), it was helpful (AP 0.764). But if they used a general speech model, it was almost useless.
Video is the Struggling Sidekick: This was a hard truth. No matter how they tried—using fancy face-scanning models, tracking eye movements, or looking at eyebrow wiggles—the video clues stayed weak. They hovered around an AP of 0.63 to 0.67. The paper suggests that with only 778 training videos, the computer just couldn't learn the subtle facial signs of hesitation. The face simply didn't give enough information.

The "Conflict" Trap: What Didn't Work

Many previous detectives thought the key to solving this was to look for conflict between the channels. They thought: "If the text says 'yes' but the face looks 'no', that's hesitation!"

The team tested this idea by building a "conflict machine" that looked for these disagreements.

The Result: It didn't help. Whether they looked for conflicts, ignored them, or split the data differently, the results barely moved (less than 0.05 change). The paper explicitly rules out "conflict design" as the secret weapon. The signal wasn't in the clash between channels; it was just in the text and the gaps.

The "Overfitting" Trap: Don't Cheat on the Test

Here is the most important lesson about how they won.

The team had a small group of 124 videos to practice on (the validation set) and a bigger group of 525 videos for the real test.

The Mistake: If you try to tune your "knobs" (weights and thresholds) specifically to get the highest score on the practice set, you end up cheating. The team tried this, and their score on the practice set was a shiny 0.741, but when they took the real test, it crashed to 0.690. This is called overfitting—memorizing the practice questions instead of learning the lesson.
The Fix: They stopped tuning the knobs. Instead, they used a simple, fixed rule: 0.5 (a 50/50 guess line) and weighted the clues based on how accurate they were historically (AP-weighting).
The Win: By sticking to this simple, honest rule, their score on the real test jumped to 0.731. This beat the previous winner (who scored 0.694).

The Final Scorecard

The team combined six different "detectives" (models) into one team.

The Result: Their final system scored a 0.731 Macro-F1 on the public test.
How sure are they? They ran a statistical check (a "bootstrap") and found that there is a 97% chance their score is actually better than the previous winner's 0.694. It's a strong lead, but not a guaranteed lock.

The Takeaway

The paper concludes that to spot hesitation, listen to the words and the silence between them, not the face.

Don't try to force the computer to find "conflict" between the eyes and the mouth; it doesn't work well here.
Do trust the text and the "ghost" time gaps left behind by speech recognition.
Do be honest with your testing: don't tweak your settings just to look good on the practice test, or you'll fail the real one.

The team's system is a simple, one-command script that proves sometimes the best way to solve a mystery isn't with a super-complex machine, but with a simple, honest look at the gaps in the story.

Technical Summary: Simple Features and Honest Calibration for Ambivalence and Hesitancy Recognition in Video

Problem Formulation
This work addresses the Ambivalence and Hesitancy (A/H) recognition task within the ABAW 2026 BAH Challenge. The objective is to predict, given a short interview video, whether the speaker exhibits signs of internal conflict or uncertainty. The task is framed as a binary, video-level classification problem evaluated primarily by macro-F1 (the unweighted mean of the F1 scores for the present and absent classes) and secondarily by Average Precision (AP). The dataset (BAH) comprises 1,427 videos from 300 participants, split into training (778), validation (124), and public test (525) sets, with no overlap in participants across splits.

Methodology
The proposed system departs from the prevailing paradigm of complex cross-modal conflict modeling, focusing instead on affect-specialized representations, interpretable linguistic cues, and rigorous calibration.

Modality Encoders: The system utilizes frozen, pre-trained encoders rather than training from scratch.
- Text: RoBERTa fine-tuned on GoEmotions (768-dim) and 11 hand-crafted hesitation markers (e.g., rates of hedges, filled pauses, self-repairs, and sentiment flips).
- Audio: An emotion-specialized wav2vec2 model (1024-dim). General speech encoders (HuBERT) were found to perform poorly.
- Video: A facial-expression Vision Transformer (FER-ViT) over 16 aligned frames, providing an embedding and a 7-way expression profile. General video encoders (VideoMAE) and specific gaze/brow dynamics failed to improve performance.
- ASR-Erased Time: A novel 16-dimensional feature set derived from the timestamps of Automatic Speech Recognition (ASR) chunks. Since ASR models (like Whisper) delete fillers and pauses to produce clean text, the temporal gaps between the remaining chunks preserve the duration of these deleted events. Features include gap counts, sizes normalized by speech time, chunk fragmentation, and pause patterns before contrastive words.
Affective Marker Fusion (AMF):
- Streams are projected to a shared width and passed through a learned reliability gate. This mechanism assigns weights to each stream, effectively turning down weak channels (like video) rather than adding them as noise.
- An auxiliary head predicts A/H using only the 11 interpretable markers, acting as a regularizer to ensure the shared representation remains predictive even when other modalities are weak.
- The model is trained with class-balanced binary cross-entropy and label smoothing ( $\epsilon=0.1$ ).
Ensemble and Calibration:
- The final prediction is an AP-weighted average of six model members (including the AMF network and individual stream probes) at a fixed decision threshold of $\tau = 0.5$ .
- The authors explicitly reject tuning the threshold or ensemble weights on the validation set, demonstrating that such "search" leads to overfitting (scoring 0.741 on validation but dropping to 0.690 on the test set).
- Instead, weights are derived solely from the Average Precision of each member, a threshold-free metric that remains stable on small datasets.
- The final model is trained on all available labelled data (train + val + test = 1,427 videos) using a seeded, stratified 113-video holdout for early stopping and weight fitting, ensuring no data leakage.

Key Results

Performance: The system achieves a 0.731 macro-F1 on the public test set, surpassing the previous edition's winner (0.694) and all baselines, including a zero-shot multi-modal LLM (0.283) and a co-attention baseline (0.590).
Signal Distribution:
- Text is dominant: The text stream (RoBERTa-GoEmotions) yields the highest AP (0.811), closely followed by the 11 hesitation markers (0.800).
- Audio is secondary: The emotion-specialized audio encoder (AP 0.764) significantly outperforms general audio encoders (HuBERT, AP 0.606).
- Video is weak: All visual representations, including FER and gaze dynamics, remain near chance levels (AP 0.63–0.67) and do not contribute meaningfully to the ensemble.
- ASR-Erased Time: This non-verbal channel achieved an AP of 0.718, outperforming all visual features and showing low correlation (0.11–0.36) with other model members, indicating high independence.
Ablation Findings:
- Conflict Design: Varying the cross-modal conflict operator (absolute difference, orthogonal split, or none) resulted in negligible changes (<0.05 AP) and inconsistent rankings between validation and test sets, proving it is not a critical lever.
- Calibration: The most significant gain came from honest calibration. Removing the validation-set search for weights and thresholds improved the test score by 0.041 (from 0.690 to 0.731).

Significance and Claims
The paper argues that for the BAH dataset, the "signal" for ambivalence and hesitancy is concentrated almost entirely in language and specific prosodic gaps, rather than in complex cross-modal facial or vocal conflict features. The authors claim three primary conclusions:

Conflict design is not the lever: Pre-designed conflict features do not reliably improve performance when encoders are fixed.
Features help narrowly: Only emotion-specialized audio and interpretable linguistic markers provide substantial gains; visual channels remain ineffective with current data scales.
Calibration is paramount: The largest performance improvement stems from avoiding overfitting during the calibration phase. Using AP-weighted ensembles with a fixed threshold is superior to tuning weights and thresholds on small validation splits.

The work emphasizes that "honest" evaluation—where calibration rules are fixed before testing—is more impactful than architectural complexity. The authors note that while the gain over the previous winner is statistically probable (95% bootstrap interval [0.693, 0.770]), it is not absolute certainty. The most promising future direction identified is the further exploitation of ASR-erased time, which currently offers a strong, independent signal that awaits larger datasets to fully convert its independence into accuracy.