Technical Summary: VISAFF

Problem Statement

Emotion Recognition in Conversation (ERC) aims to identify the emotional state of speakers in multi-turn dialogues. While early text-based methods are straightforward, they fail in complex scenarios (e.g., sarcasm) by neglecting vital non-verbal information like facial expressions and body gestures. Recent approaches utilizing Vision-Language Models (VLMs) attempt to analyze video directly but face three critical limitations:

1. Lack of Speaker Focus: General VLMs are not inherently tailored for ERC; they often attend to visually salient but emotionally irrelevant regions (e.g., background, passive listeners) rather than the active speaker.
2. Computational Cost: Fine-tuning large VLMs or using parameter-efficient adaptation (e.g., LoRA) for ERC incurs prohibitive computational costs and requires extensive annotated resources.
3. Visual Ambiguity: Isolated visual signals are frequently ambiguous or technically compromised (e.g., motion blur) without the context of linguistic content and vocal prosody. A smile, for instance, may not indicate happiness without conversational context.

The paper addresses two core challenges: (1) How to unlock the latent ERC reasoning capabilities of frozen VLMs using a tuning-free approach to focus on the active speaker's emotional cues; and (2) How to design an adaptive mechanism to dynamically leverage textual and acoustic modalities to compensate for visual uncertainty.

Methodology: VISAFF

The authors propose VISAFF (Speaker-Centered VISual AFFective feature learning), a two-stage framework that operates without fine-tuning the underlying VLM.

Stage 1: Speaker-Centered Affective Grounding (SCAG)

This stage utilizes a frozen VLM as a zero-shot inference-time feature extractor. To guide the VLM effectively without training, the authors introduce two input groups:

• Prompt-Guided VLM Inputs (PGVI): This includes sampled video frames ($F_i$), a target speaker reference image ($R_i$) to serve as a localization constraint, and a task prompt ($P_{task,i}$) that explicitly instructs the model to perform speaker-centered affective analysis rather than generic scene description.
• Affective Semantic Guidance Inputs (ASGI): To provide necessary emotional context that isolated frames lack, this input concatenates prompts derived from dialogue context ($P_{ctx}$), audio descriptions ($P_{aud}$), and lexical Valence-Arousal-Dominance (VAD) priors ($P_{vad}$).

The frozen VLM processes these combined inputs to extract speaker-centered visual affective features ($v_i$). No gradients are back-propagated to the VLM parameters.

Stage 2: Reliability-Guided Affective Complementation (RGAC)

Recognizing that visual cues can be unreliable, this stage adaptively integrates textual and acoustic modalities based on visual reliability.

• Visual-Guided Cross-Modal Retrieval: Textual (RoBERTa) and acoustic (emotion2vec) features are projected into a shared space. Using the visual feature as a query, the model retrieves textual and acoustic affective references ($\tilde{h}_i^{(m)}$) via cross-attention. These are treated as external references to interpret the visual cues, not as independent dominant modalities.
• Reliability-Gated Residual Complementation: A visual reliability score ($c_i$) is computed (via an auxiliary visual classifier). The model calculates a residual complement ($\Delta_i$) representing the discrepancy between the retrieved external references and the current visual state. The final visual representation is updated as:
  $$h_i^{(v)*} = h_i^{(v)} + (1 - c_i)\Delta_i$$
  When visual cues are reliable ($c_i \approx 1$), the external complement is suppressed. When visual cues are uncertain ($c_i \approx 0$), textual and acoustic references provide stronger compensation.
• Prediction: The complemented visual representation is concatenated with the retrieved text and audio references for final emotion classification.

Training Objectives

The framework is optimized using three losses:

1. Classification Loss ($L_{cls}$): Supervises the final emotion prediction.
2. Auxiliary Visual Loss ($L_{aux}$): Supervises a video-only classifier to stabilize the visual reliability score and preserve the visual branch's discriminative ability.
3. Semantic Alignment Loss ($L_{cl}$): Regularizes the multimodal feature space using InfoNCE and supervised contrastive learning before cross-modal interaction.

The paper provides a theoretical generalization error bound (Theorem 1) supporting the reliability-guided design, showing that the upper bound tightens when visual confidence is negatively correlated with visual loss and positively correlated with the utility of external modalities.

Key Contributions

1. VISAFF Framework: A speaker-centered visual affective feature learning framework that activates frozen VLMs for visual reasoning without ERC-specific fine-tuning, significantly reducing computational overhead.
2. SCAG Mechanism: A zero-shot grounding method using Prompt-Guided VLM Inputs and Affective Semantic Guidance to steer frozen VLMs toward the active speaker's emotional visual cues.
3. RGAC Mechanism: A reliability-guided affective complementation module that adaptively integrates textual and acoustic information to compensate for visual uncertainty, controlled by a learned visual reliability gate.
4. Empirical Validation: Extensive experiments on two real-world datasets (MELD and IEMOCAP) demonstrating competitive performance against state-of-the-art methods, including those using fine-tuned large models.

Experimental Results

Experiments were conducted on the MELD and IEMOCAP datasets using Qwen3-VL-Embedding as the frozen backbone.

• Performance: VISAFF achieved the best Weighted F1 (W-F1) scores among traditional ERC methods and large-model-based methods that do not use fine-tuning or LoRA adaptation.
  • IEMOCAP: 77.30% W-F1 (vs. 63.78% for DialogueLLM w/o LoRA).
  • MELD: 67.12% W-F1 (vs. 64.42% for DialogueLLM w/o LoRA).
• Ablation Studies:
  • Removing SCAG (feeding raw frames to VLM) caused a significant performance drop, confirming the necessity of speaker grounding and semantic guidance.
  • Removing RGAC led to a clear decline, validating the importance of reliability-guided complementation.
  • Component analysis showed that both textual and acoustic references, as well as the reliability gate, contribute positively to the final prediction.
• Reliability Analysis: The framework showed the most significant performance gains in low-confidence visual bins (e.g., +24.1% gain on IEMOCAP for the 0.2–0.4 confidence bin), demonstrating that the model effectively leverages text/audio when visual cues are ambiguous.

Significance and Claims

The paper claims that VISAFF provides a highly efficient framework for multimodal reasoning in dynamic dialogue scenarios. Its primary significance lies in demonstrating that:

1. Frozen VLMs are viable for ERC: With proper task-guided grounding and semantic prompting, frozen VLMs can extract high-quality speaker-centered visual affective features without the prohibitive cost of fine-tuning.
2. Speaker-Centered Focus is Critical: Explicitly guiding the model to the active speaker and their emotional cues, rather than relying on global video features, is essential for accurate ERC.
3. Contextual Reliability Matters: Visual signals are inherently ambiguous; a reliability-aware mechanism that dynamically supplements visual uncertainty with linguistic and acoustic context significantly enhances robustness.

The authors position this work as a visual-oriented perspective for ERC, encouraging future research into speaker-centered visual reasoning and reliability-aware multimodal complementation.