Technical Summary: PRISM-AH

Problem Definition

The paper addresses the challenge of recognizing Ambivalence and Hesitancy (A/H) in video-based health behavior change interventions. A/H represents a conflicted affective state between positive and negative orientations toward an action, often leading to the delay or abandonment of behavior change.

The core difficulty lies in the nature of the evidence:

• Sparsity and Localization: Discriminative cues are sparse and localized in time rather than distributed uniformly across a video.
• Cross-Channel Disagreement: The most reliable evidence often arises from conflicts between modalities (e.g., verbal affirmation accompanied by an averted gaze) rather than strong signals within a single channel.
• Individual Variability: Individuals exhibit hesitation differently, and the annotated corpus (the Behavioural Ambivalence/Hesitancy or BAH benchmark) is modest in size.
• Limitations of Standard Fusion: Direct application of standard multimodal fusion to dense per-frame features yields low accuracy because pooling features dilutes temporal locality and fails to capture cross-channel disagreement.

Methodology: PRISM-AH

The proposed framework, PRISM-AH (Predictive Reasoning over Interacting Streams for Multimodal Ambivalence / Hesitancy recognition), departs from standard feature fusion. Instead, it reformulates the task as knowledge-guided reasoning over structured, timestamped multimodal evidence. The architecture consists of two primary stages:

1. Structured Evidence Extraction (Discriminative Network)

A lightweight multimodal network processes video windows (0.5 seconds) to generate a compact, interpretable summary rather than a single pooled vector.

• Encoders: Frozen foundation encoders are used for each modality:
  • Vision: CLIP-family ViT for face frames.
  • Audio: WavLM for speech.
  • Text: MPNet for transcripts.
  • Prosody/Disfluency: Explicit descriptors (pitch, energy, pauses, hedges, etc.) are computed per window.
• Interaction Hub: Channels are projected to a shared dimension and modulated by participant metadata via Feature-wise Linear Modulation (FiLM). A stack of interaction blocks uses cross-attention and temporal self-attention to couple channels.
• Signal Generation: The network emits specific signals to enrich the evidence:
  • Conflict Score: Explicitly measures cross-channel disagreement via pairwise cosine dissonance ($d_{va}, d_{v\tau}, d_{a\tau}$).
  • Hesitation Surprise: A predictive component (inspired by JEPA) anticipates the next hub state; prediction errors signal breaks in behavioral continuity.
  • Prototype Matching: A bank of learnable prototypes assigns interpretable pattern labels to windows.
• Output: The network produces a video-level discriminative probability and a structured evidence summary containing timestamps, dominant modalities, conflict scores, and transcript segments.

2. Knowledge-Guided Reasoning and Late Fusion

A Large Language Model (LLM) acts as a reasoner, operating in a zero-shot manner on the structured evidence.

• Prompting: The LLM receives the evidence summary under a prompt that explicitly encodes the expert cue taxonomy of the BAH benchmark. This provides the domain knowledge necessary to interpret subtle cues.
• Verdict: The LLM returns a verdict with a confidence score and a rationale citing decisive moments.
• Fusion: The reasoning probability ($q$) is combined with the discriminative probability ($p$) via convex late fusion: $p_{final} = (1 - \omega)p + \omega q$. The weight $\omega$ and threshold are calibrated on the validation partition to maximize macro F1.

Key Contributions

1. Reformulation of the Task: The work shifts from dense feature fusion to reasoning over structured, timestamped evidence, achieving a macro F1 of 0.6324 on the private test partition (a 2.24× improvement over the zero-shot baseline of 0.2827).
2. Knowledge-Guided Reasoning Layer: The introduction of an LLM reasoner that interprets evidence under an explicit encoding of the expert cue taxonomy. Controlled studies confirm that the encoded taxonomy is the essential ingredient for performance gains; removing it eliminates the improvement.
3. Rigorous Empirical Analysis: A controlled study over three seeds quantifies modality contributions and robustness. It identifies language as the primary driver of predictive performance and demonstrates that conditioning on participant attributes does not transfer to participant-disjoint partitions.

Experimental Results

• Performance: On the private test partition, PRISM-AH achieves a macro F1 of 0.6324. On the public test partition, the discriminative network alone reaches 0.5790, which improves to 0.6133 with the addition of knowledge-guided reasoning.
• Reasoning Layer Necessity:
  • Model Capacity: A 7B parameter model yielded no improvement (fusion weight = 0), while a 14B model raised the score to 0.6070, indicating a need for sufficient reasoning capacity.
  • Prompt Ablation: Removing the expert cue taxonomy from the prompt eliminated the gain (returning to 0.5825), whereas removing the structured evidence retained the gain (0.6147). This suggests the domain knowledge is more critical than the specific evidence format for the reasoner.
• Modality Analysis:
  • Single-modality models (Vision, Audio, Text) performed comparably (macro F1 ~0.55–0.56).
  • Language Dominance: Removing the language channel caused the largest performance degradation (0.5735 $\to$ 0.4982), confirming text as the decisive channel.
  • Participant Conditioning: Removing metadata conditioning actually improved the public test score (0.5570 $\to$ 0.6140), indicating that participant attributes fit the training distribution but do not generalize to unseen individuals.
• Robustness: The system showed narrow performance gaps across gender and age subgroups.

Significance and Claims

The paper claims that the primary source of performance improvement is the knowledge-guided reasoning layer, specifically the explicit encoding of the expert cue taxonomy. The study suggests that for tasks where evidence is sparse, localized, and defined by cross-modal conflict, distilling video into structured evidence and applying domain-grounded reasoning is more effective than fusing dense features.

The authors conclude that a compact discriminative backbone is preferable, as auxiliary structural signals (conflict scores, surprise signals) contributed marginally in the discriminative-only setting. The most promising direction for future improvement is identified as stronger language grounding, consistent with the finding that language is the decisive channel for A/H recognition. The approach avoids over-reliance on participant-specific conditioning, favoring a configuration that generalizes to unseen individuals.