Ambivalence/Hesitancy Recognition in Videos for Personalized Digital Health Interventions

1. Problem Statement

The paper addresses the critical challenge of Ambivalence and Hesitancy (A/H) in digital health interventions.

• Context: Health interventions aim to change high-risk behaviors (e.g., smoking, inactivity) to prevent chronic diseases. However, individuals often delay, avoid, or abandon these changes due to A/H—a state of conflicting emotions between willingness and resistance.
• The Gap: While human clinicians can detect A/H through subtle verbal and non-verbal cues (facial expressions, tone, body language) during in-person interactions, digital health interventions lack this capability. Current automated emotion recognition systems focus on basic discrete emotions (happy, sad) or continuous dimensions (valence/arousal), failing to capture the compound, conflicting nature of A/H.
• Challenge: A/H is inherently multimodal and subtle. It manifests as affective inconsistency within a single modality (e.g., a contradictory tone) or across modalities (e.g., a smiling face saying "I don't want to quit"). Detecting this requires sophisticated modeling of spatio-temporal dynamics and cross-modal conflicts, which current off-the-shelf models struggle to perform.

2. Methodology

The authors evaluate three distinct learning scenarios using the BAH (Behavioural Ambivalence/Hesitancy) dataset, which contains 1,427 videos from 300 participants with visual, audio, and text modalities.

A. Data Pre-processing

• Visual: Faces are cropped and aligned using RetinaFace; resized to 256×256.
• Audio: Converted to mono (16kHz); log-Mel spectrograms extracted via VGGish.
• Text: Audio transcribed using Whisper; word-level features extracted via BERT. Features are synchronized with video frames.

B. Learning Scenarios

1. Supervised Learning (Frame & Video Level):

   • Backbones: Tested various CNNs (ResNet18-152) and Vision Transformers (APViT). Models were pre-trained on basic emotion datasets (RAF-DB, AffectNet, Aff-wild2) before fine-tuning on BAH.
   • Context: Compared independent frame classification vs. temporal modeling using Temporal Convolutional Networks (TCN).
   • Multimodal Fusion: Evaluated single modalities (Visual, Audio, Text) and combinations. Tested fusion techniques: Concatenation (CAN), Co-attention (LFAN), Transformer (MT), and Cross-attention (JMT).
   • Video Classification: Utilized Video-FocalNet and EmoCLIP (CLIP-based) for video-level classification.

2. Personalization via Unsupervised Domain Adaptation (UDA):

   • Goal: Adapt a source-trained model to individual target users (treated as distinct domains) without labeled target data.
   • Methods:
     • UDA: MMD (Maximum Mean Discrepancy) and ACPL (combining pseudo-labeling and MMD).
     • Source-Free DA (SFDA): SHOT and NRC (Neighborhood Reciprocity Consistency).
     • Test-Time Adaptation (TTA): Methods like TPT, TDA, and CLIP-AUTT that adapt at inference time using only unlabeled target data.

3. Zero-Shot Inference with Multimodal LLMs (MLLMs):

   • Models: Video-LLaVA-7B and AffectGPT.
   • Strategy: Tested various prompt engineering variations (Simple, Definition-based, Transcript-based) to infer A/H without fine-tuning.
   • Inference: Used a sliding window approach for frame-level prediction and uniform sampling for video-level prediction.

3. Key Contributions

• Task Definition: Formalizes A/H Recognition as a distinct, challenging multimodal task in digital health, moving beyond basic emotion recognition.
• Comprehensive Benchmarking: Provides the first extensive evaluation of deep learning models (CNNs, ViTs, LLMs) on the BAH dataset across supervised, domain adaptation, and zero-shot settings.
• Empirical Insights:
  • Demonstrates that textual transcripts are the most informative single modality for A/H detection.
  • Shows that standard fusion techniques (concatenation) often underperform compared to attention-based mechanisms, though all methods struggle with the subtle nature of A/H.
  • Highlights that context (temporal modeling) is crucial, as A/H rarely occurs in a single frame.
• Framework Proposal: Proposes a novel two-level framework for future work:
  1. Independent modeling of affect per modality.
  2. A dedicated fusion mechanism specifically designed to detect affective conflicts (dissonance) across modalities, rather than just fusing features.

4. Results

• Supervised Learning:
  • Performance: Results are limited. The best frame-level AVGF1 (Average F1) was 0.5586 (Audio + Text), and video-level AVGF1 reached 0.5663 (Video-FocalNet Base).
  • Modality: Text alone outperformed visual and audio. Combining all three modalities did not significantly improve performance, suggesting current fusion methods fail to capture cross-modal conflicts effectively.
  • Context: Adding temporal context (TCN) improved performance across all architectures, confirming the importance of sequence modeling.
• Domain Adaptation:
  • UDA: ACPL achieved the best results (AVGF1: 0.5417), outperforming Source-only (0.4894).
  • TTA: While TTA methods improved Weighted Average Recall (WAR), gains in AVGF1 were limited, indicating difficulty in handling class imbalance without source data.
  • Oracle: The upper bound (fine-tuned on target data) was significantly higher (AVGF1: 0.5864), indicating room for improvement in unsupervised personalization.
• Zero-Shot (LLMs):
  • Prompt Sensitivity: Performance was highly dependent on prompts. Providing the video transcript and a clear definition of A/H yielded the best results (Video-LLaVA AVGF1: 0.6341).
  • Fine-tuning: Simple fine-tuning improved results to 0.7142, but the authors note that specialized modeling is still required to fully capture the complexity of A/H.

5. Significance and Future Directions

• Clinical Impact: Accurate A/H recognition is vital for personalizing digital health interventions. By detecting hesitation, apps can dynamically adjust strategies (e.g., offering encouragement vs. information) to prevent user drop-off.
• Technical Limitations: The paper concludes that existing state-of-the-art models are insufficient for A/H recognition. The subtle, conflicting nature of the emotion requires models specifically designed to detect inconsistencies rather than just aligning features.
• Future Work Recommendations:
  • Develop specialized conflict-detection modules that explicitly model dissonance between modalities.
  • Leverage annotator statistics to constrain models to find specific types of conflicts.
  • Explore textualizing audio and visual cues to better leverage the reasoning capabilities of Large Language Models for this task.
  • Investigate segmented body analysis to reduce noise compared to full-frame analysis.

In summary, this paper establishes a baseline for A/H recognition in digital health, demonstrating that while current deep learning and LLM approaches show promise, significant architectural innovations are needed to effectively model the complex, conflicting affective states that hinder behavior change.