Conflict-Aware Multimodal Fusion for Ambivalence and Hesitancy Recognition

1. Problem Definition

The paper addresses the automatic recognition of Ambivalence and Hesitancy (A/H) in clinical interview videos.

• Nature of the Problem: A/H represents a subtle affective state where an individual holds conflicting feelings or delays decision-making. Unlike basic emotions (e.g., happiness, anger) which often manifest consistently across modalities, A/H is characterized by cross-modal disagreement (e.g., a patient verbally agreeing to a treatment while their facial expression or voice tone indicates doubt or stress).
• Challenge: Standard multimodal fusion methods typically seek consensus among modalities, often suppressing the very disagreement signals that define A/H. Furthermore, existing approaches are heavily text-dominant, leading to a systematic bias: they over-detect A/H (high precision on positive cases) but struggle to confirm its absence (low recall on negative cases), resulting in a large performance gap between classes.
• Dataset: The work utilizes the BAH dataset from the ABAW10 Challenge, containing 1,427 clinical interview videos (778 for training). The dataset is small and expert-annotated, requiring efficient regularization and pre-trained representations.

2. Methodology: ConflictAwareAH

The authors propose ConflictAwareAH, a hybrid multimodal framework designed to explicitly model cross-modal conflicts.

A. Architecture Overview

The pipeline consists of three pre-trained encoders, an attention pooling mechanism, a conflict-aware fusion module, and a text-guided late fusion strategy.

1. Multimodal Encoders:

   • Video: VideoMAE-Base (pre-trained on Kinetics-400) processes 16-frame face crops.
   • Audio: HuBERT-Base (fine-tuned on LibriSpeech) encodes raw 16kHz audio to capture prosodic cues (pauses, pitch variation).
   • Text: RoBERTa-Base (fine-tuned on GoEmotions) encodes the full video transcript. The full transcript is used rather than window-aligned text because hesitation cues can occur anywhere in the interview.
   • Output: All encoders produce 768-dimensional embeddings.

2. Attention Pooling:

   • Variable-length sequences from encoders are reduced to fixed-size vectors using learnable soft-attention pooling. This allows the model to weigh emotionally salient moments (e.g., a hesitation pause) more heavily than uninformative tokens.

3. Conflict-Aware Fusion (Core Innovation):

   • Instead of concatenating modalities directly, the model computes pairwise absolute differences between embeddings: $|v - a|$, $|v - t|$, and $|a - t|$.
   • Logic: Large differences indicate cross-modal disagreement (flagging A/H), while small differences indicate consistency (anchoring the negative class).
   • The final fusion vector concatenates the modality embeddings and the three conflict vectors: $[v; a; t; c_{va}; c_{vt}; c_{at}]$.
   • This vector is processed by a two-layer Feed-Forward Network (FFN) to produce the full-fusion logit ($\ell_{full}$).

4. Text-Guided Late Fusion:

   • Recognizing that text is the strongest individual signal, a parallel text-only auxiliary head ($\ell_{text}$) is trained alongside the full model.
   • Inference Strategy: The final prediction is a weighted blend: $P = \alpha \cdot \sigma(\ell_{text}) + (1-\alpha) \cdot \sigma(\ell_{full})$, with $\alpha=0.6$.
   • Purpose: This acts as a regularizer, preventing the multimodal path from overfitting to noisy visual/audio patterns while leveraging the strong linguistic signal for ambiguous cases.

B. Training Protocol

• Optimization: AdamW with cosine annealing.
• Regularization: Given the small dataset (778 samples), encoder weights are mostly frozen (top-2 layers optionally unfrozen). Label smoothing and class weighting are used to handle imbalance.
• Efficiency: The model trains in under 25 minutes on a single NVIDIA RTX 4090 GPU.
• Ensembling: The final submission averages predictions from two checkpoints with different regularization settings and uses multi-window inference (averaging 5 random 16-frame windows).

3. Key Contributions

1. Conflict-Aware Fusion: Introduced explicit pairwise conflict features (absolute differences) as bidirectional cues. This corrects the class bias of text-only models, significantly improving the detection of non-A/H cases.
2. Text-Guided Late Fusion: A strategy that blends a text-only auxiliary head with the multimodal output, boosting Macro F1 by leveraging the dominant linguistic signal while retaining multimodal robustness.
3. Efficient Baseline: Achieved state-of-the-art results with a lightweight pipeline that requires minimal training time and computational resources compared to large-scale fine-tuning approaches.

4. Experimental Results

Evaluated on the ABAW10 Ambivalence/Hesitancy Challenge:

• Performance:
  • Labelled Test Split: 0.694 Macro F1 (Ensemble).
  • Private Leaderboard: 0.715 Macro F1.
  • Comparison: Outperforms the best published multimodal baseline (LFAN) by >10 points (0.593 vs. 0.694) and the zero-shot Multimodal LLM with transcript by ~6 points (0.634 vs. 0.694).
• Ablation Insights:
  • Text Dominance: Text alone achieves 0.737 Macro F1 on validation but suffers from a large class-performance gap (F1-AH: 0.804 vs. F1-NoAH: 0.670).
  • Conflict Features: Adding conflict features improves F1-NoAH by +4.6 points over text-only, effectively halving the class-performance gap. This confirms that cross-modal disagreement is a critical signal for A/H.
  • Late Fusion: The text-guided blend adds +4.1 Macro F1 over the full-fusion-only model.

5. Significance and Clinical Impact

• Clinical Relevance: In clinical settings (e.g., motivational interviewing), false positives (flagging a patient as hesitant when they are not) waste clinician time. The proposed model's ability to confirm the absence of ambivalence (high F1-NoAH) is as valuable as detecting its presence.
• Paradigm Shift: The paper demonstrates that for complex affective states like A/H, disagreement between modalities is a more informative feature than consensus.
• Efficiency: The method proves that high-performance affective computing can be achieved with small datasets and single-GPU training by leveraging pre-trained representations and smart fusion strategies, making it more accessible for real-world deployment.

Conclusion

ConflictAwareAH successfully reframes A/H recognition as a problem of detecting cross-modal inconsistency. By explicitly modeling conflicts and balancing text dominance with multimodal consistency checks, the authors achieved a new state-of-the-art on the BAH dataset, offering a robust, efficient, and clinically viable solution for detecting subtle human affective states.