BAH Dataset for Ambivalence/Hesitancy Recognition in Videos for Digital Behavioural Change

This paper introduces the BAH dataset, a multimodal collection of 1,427 videos from 300 participants annotated for ambivalence and hesitancy recognition, alongside baseline benchmarking results that highlight the need for advanced models to support personalized digital health interventions.

The Gist

Imagine you are trying to convince a friend to start a new habit, like jogging every morning. They say, "I really want to do it," but their voice sounds shaky, they look away, and they keep fidgeting. They aren't saying "no," but they aren't fully saying "yes" either. They are stuck in the middle, torn between wanting to change and being afraid to do so.

In psychology, this feeling is called Ambivalence or Hesitancy. It's the "maybe" state. It's the most common reason people quit health goals before they even start.

This paper introduces a new tool to help computers understand that tricky "maybe" feeling. Here is the breakdown of what the researchers did, explained simply:

1. The Problem: Computers Are Bad at Reading "Maybe"

Right now, computers are great at spotting obvious emotions. If someone is laughing, the computer knows it's "Happy." If they are crying, it knows it's "Sad."

But Ambivalence is different. It's a silent conflict. It's when your face says "I'm excited," but your voice sounds tired, and your body language says "I'm scared." It's like a radio playing two different stations at once. Humans are good at spotting this because we are trained to read subtle social cues. Computers, however, have never seen a dataset designed specifically to teach them how to spot this "conflict." They usually just guess, and they guess wrong.

2. The Solution: The "BAH" Dataset

To fix this, the researchers created a massive new library of videos called the BAH Dataset (Behavioural Ambivalence/Hesitancy).

• The Cast: They recruited 300 real people from across Canada.
• The Script: These people sat in front of their webcams and answered seven questions designed to make them feel conflicted.
  • Example Question: "Tell us about something you enjoy doing but wish you stopped." (This forces the brain to juggle "I like this" vs. "I should stop").
• The Result: They captured 1,427 videos. Some people were clearly happy, some were clearly resistant, but many were stuck in that messy middle ground of ambivalence.

Think of this dataset as a gym for AI. Just as a weightlifter needs heavy weights to get strong, an AI needs thousands of examples of "conflicted" humans to learn how to spot them.

3. The "Human Coaches" (Annotation)

You can't just feed raw video to a computer; you have to tell it what to look for. The researchers hired three experts in human behavior to watch every single video.

They acted like detectives, looking for tiny clues:

• The Eyes: Did they look away when talking about a goal?
• The Voice: Did they pause too long or sound shaky?
• The Body: Did they shrug or cross their arms?
• The Conflict: Did the person say "Yes" while shaking their head "No"?

The experts marked exactly when these moments happened, frame by frame. They created a "cheat sheet" for the computer, teaching it that a specific combination of a shaky voice + averted gaze = Hesitancy.

4. The Training: Teaching the AI

The researchers then tried to teach a computer to recognize these moments using their new dataset.

• The First Attempt: They used standard AI models (the ones that usually spot "Happy" or "Sad").
• The Result: The computer struggled. It was like trying to teach a dog to play chess. The computer got about 50-55% accuracy. It was barely better than flipping a coin.
• Why? Because Ambivalence is subtle. It's not a loud explosion; it's a whisper. The computer needed to look at the whole picture (face + voice + words) and understand the timing (how the emotion changed over a few seconds).

5. What This Means for the Future

Why do we care? Imagine a Digital Health Coach (like a chatbot or a virtual trainer) that talks to you every day.

• Without this tech: The bot asks, "Did you go for a run?" You say, "I tried," but you sound unsure. The bot just says, "Great job!" and misses the fact that you are actually struggling and about to quit.
• With this tech: The bot notices your hesitation. It sees the conflict in your voice and face. It realizes you are ambivalent. Instead of just saying "Good job," it might say, "It sounds like you're having a tough time. What's holding you back? Let's figure this out together."

The Big Takeaway

This paper is the foundation for building smarter, more empathetic digital health tools. By creating a library of "conflicted" human moments, the researchers have given AI the first real chance to understand that humans aren't just "Yes" or "No." We are often stuck in the middle, and now, for the first time, computers have a map to help us navigate that middle ground.

In short: They built a library of "maybe" moments so computers can finally learn to understand when we are torn, helping us stick to our health goals with better support.

Technical Summary

1. Problem Statement

The paper addresses the critical challenge of Ambivalence and Hesitancy (A/H) in health-related behavior change. A/H represents a state of conflicting emotions where an individual oscillates between positive and negative orientations (e.g., willingness vs. resistance). These subtle, conflicting states are primary reasons individuals delay or abandon health interventions.

• Current Limitations: While human experts can detect A/H through verbal and non-verbal cues in face-to-face interactions, integrating this capability into digital health interventions is costly and inefficient.
• ML Gap: Existing machine learning (ML) research focuses heavily on basic discrete emotions (e.g., happy, sad) or continuous dimensions (valence/arousal). There is a lack of specialized datasets and models for recognizing compound, conflicting emotions like A/H, which manifest as discord within a single modality (e.g., a hesitant tone) or between modalities (e.g., saying "yes" while shaking the head).
• Data Scarcity: No existing dataset supports the training of ML models for A/H recognition in video, hindering the development of personalized, automated digital health tools.

2. Methodology: The BAH Dataset

The authors introduce the Behavioural Ambivalence/Hesitancy (BAH) dataset, the first multimodal video dataset specifically designed for A/H recognition.

Data Collection

• Participants: 300 participants recruited across 9 Canadian provinces to ensure demographic diversity (age, ethnicity, gender).
• Protocol: Participants used a web-based platform ("Automatic Expression Recognition") to record themselves answering 7 structured questions designed to elicit A/H (e.g., "Tell us about something you enjoy but wish you stopped").
• Scale: The dataset comprises 1,427 videos (totaling ~10.60 hours) with 916,618 frames.
• Content: 778 videos contain A/H instances (1.79 hours), totaling 1,504 A/H segments. The average A/H segment is brief (4.29 seconds or ~104 frames).

Annotation Strategy

• Experts: Three behavioral science experts annotated the data.
• Levels:
  • Video-level: Binary label (A/H present vs. absent).
  • Frame-level: Precise timestamps for the onset and offset of A/H segments.
• Cues: Annotators identified specific cues across four modalities: Facial expressions, Body language, Audio (vocal), and Language (transcript). Crucially, they also flagged cross-modal inconsistencies (e.g., facial expression contradicting speech).
• Format: The dataset includes raw videos, cropped/aligned faces, audio transcripts, and participant metadata.

Experimental Setup

• Split: Randomly divided by participant into Train (195), Validation (30), and Test (75) sets to prevent data leakage.
• Imbalance: The dataset is highly imbalanced at the frame level (only ~17% of frames contain A/H).
• Metrics: Due to imbalance, performance is evaluated using Average F1 (AVGF1) and Average Precision (AP) for the positive class.

3. Key Contributions

1. Novel Dataset (BAH): A large-scale, multimodal video dataset for A/H recognition, featuring frame-level annotations, temporal localization, and detailed behavioral cues.
2. Benchmarking: Comprehensive baseline results using state-of-the-art models for:
   • Frame-level and video-level classification.
   • Zero-shot inference using Multimodal Large Language Models (M-LLMs).
   • Personalization via Source-Free Domain Adaptation (SFUDA).
3. Insights into A/H: Analysis revealing that A/H is inherently multimodal and temporal. The study highlights that standard fusion methods struggle with the "conflict" nature of A/H, necessitating specialized architectures.
4. Open Resources: Public release of the dataset, code, and pretrained weights to foster research in digital behavioral health.

4. Results and Findings

Baseline Performance

• Difficulty: Baseline models achieved modest performance (e.g., AVGF1 ~0.55, AP ~0.26), confirming that A/H is a significantly harder task than basic emotion recognition.
• Modality Impact:
  • Text alone performed best among single modalities (AP 0.2519), suggesting verbal cues are strong indicators.
  • Multimodal Fusion: Combining Audio + Text yielded the best pair performance (AVGF1 0.5586). However, adding Visual modality to all three slightly reduced AP, indicating that naive fusion fails to capture the conflict between modalities.
• Temporal Context: Using temporal context (via TCN or sliding windows) significantly improved performance over single-frame analysis, as A/H is a sustained state, not an instantaneous event.

Advanced Experiments

• Zero-Shot (M-LLMs): Using Video-LLaVA, zero-shot performance was highly dependent on prompts. Providing the video transcript and a definition of A/H significantly boosted performance (Video-level AVGF1 reached 0.6341), reinforcing the importance of textual context.
• Personalization (Domain Adaptation):
  • Standard "Source-only" models struggled to generalize to new individuals.
  • Subject-based Domain Adaptation (Sub-based) using pseudo-labeling achieved the best results (AVGF1 0.5417), outperforming other UDA/SFUDA methods. This suggests that personalizing models to individual users is critical for A/H detection.

Key Observations

• Conflict Detection: Standard fusion learns to align modalities. A/H requires detecting misalignment (conflict). The authors propose a two-level framework: independent modality modeling followed by a dedicated conflict-detection fusion layer.
• Temporal Nature: A/H segments are short and fluctuating. Large context windows can degrade performance, suggesting an optimal window size exists (around 1-5 seconds).

5. Significance and Future Directions

• Digital Health Impact: The BAH dataset enables the development of automated systems that can detect user hesitation in real-time digital interventions (e.g., virtual coaches, therapy apps). This allows systems to adapt their strategy when a user is ambivalent, potentially improving adherence and health outcomes.
• Research Direction: The paper shifts the focus from basic emotion recognition to compound, conflicting emotions. It establishes that future models must be:
  1. Multimodal: Integrating visual, audio, and text.
  2. Conflict-Aware: Specifically designed to detect inconsistencies between modalities.
  3. Personalized: Capable of adapting to individual user baselines via domain adaptation.
• Availability: The dataset and code are publicly available, serving as a foundational resource for the affective computing and digital health communities.

In summary, this paper fills a critical gap in affective computing by providing the necessary data and benchmarks to tackle the complex, subtle, and multimodal nature of ambivalence and hesitancy, paving the way for more effective and personalized digital health interventions.