Nuanced Emotion Recognition Based on a Segment-based MLLM Framework Leveraging Qwen3-Omni for AH Detection

1. Problem Statement

The paper addresses the challenge of recognizing Ambivalence and Hesitancy (A/H) in video data. Unlike basic emotions (e.g., happiness, sadness), A/H represents a subtle, conflicting psychological state where an individual simultaneously holds positive and negative attitudes.

• Key Challenges:
  • Cross-modal Inconsistencies: A/H often manifests as discrepancies between modalities (e.g., positive verbal content delivered with a hesitant tone or fluctuating facial expressions).
  • Temporal Complexity: A/H signals are typically short-lived (average duration ~4.29 seconds) but embedded within longer videos.
  • Computational Constraints: Feeding long videos directly into Multimodal Large Language Models (MLLMs) exceeds token limits, increases computational costs, and introduces irrelevant noise.
• Dataset: The study utilizes the BAH dataset, containing 1,427 videos (10.6 hours) from 300 participants with expert annotations at both video and frame levels.

2. Methodology

The proposed framework integrates temporal segmentation with a powerful MLLM to detect A/H. The pipeline consists of three main stages:

A. A/H-Aware Video Preprocessing

The system leverages the BAH dataset's detailed timestamp annotations to construct high-density training samples:

• Negative Samples: Videos without A/H are kept as single units.
• Positive Samples: Only the specific temporal segments annotated as containing A/H are extracted. This removes irrelevant context and focuses the model on the signal of interest.

B. Clip-Based Temporal Segmentation

To address token limits and computational efficiency, the authors propose a 5-second clip strategy:

• Videos (or extracted A/H segments) are divided into clips with a maximum duration of 5 seconds.
• Clips shorter than 5 seconds are kept intact; longer segments are split into consecutive 5-second windows.
• Trailing segments that are too short are discarded.
• Rationale: This aligns with the average duration of A/H signals, ensuring the model captures local emotional cues without exceeding the MLLM's context window.

C. Multimodal Large Model Adaptation

The core engine is Qwen3-Omni-30B-A3B, a multimodal LLM capable of processing video, audio, and text simultaneously.

• Fine-tuning: The model is fine-tuned using the MS-Swift framework. Two strategies are employed:
  1. LoRA (Low-Rank Adaptation): Parameter-efficient tuning.
  2. Full-Parameter Fine-tuning: Updating all model weights.
• Task Formulation: The task is framed as a binary reasoning question. The model receives video frames, audio features, and a prompt, outputting <answer>Yes</answer> or <answer>No</answer>.

D. Inference and Aggregation

• Clip-Level Prediction: The model predicts A/H presence for each 5-second clip.
• Video-Level Aggregation: A Max-based rule is applied: if any clip within a video is predicted as positive, the entire video is classified as positive ($y_{video} = \max(y_i)$).
• Ensemble Strategy: To improve robustness, predictions from multiple models (trained with different configurations) are combined using Majority Voting.

3. Key Contributions

1. Segment-Based Temporal Modeling: A novel strategy that segments videos into 5-second clips. This respects MLLM token constraints while focusing on the specific temporal windows where A/H signals occur.
2. MLLM Adaptation for A/H: Successful adaptation of the Qwen3-Omni-30B-A3B model to detect subtle cross-modal emotional conflicts, demonstrating the capability of large models to reason about nuanced psychological states.
3. Robust Ensemble Framework: The combination of clip-level max-aggregation and multi-model majority voting significantly reduces prediction variance, particularly for ambiguous samples.

4. Experimental Results

Experiments were conducted on the BAH test set using the MS-Swift framework with bfloat16 precision and FlashAttention.

• Performance: The proposed ensemble method achieved an accuracy of 85.1%.
• Comparison:
  • Qwen3-Omni-v1 (LoRA, 1 epoch): 81.9%
  • Qwen3-Omni-v2 (Full FT, 2 epochs): 79.8%
  • Qwen3-Omni-v3 (Full FT, 3 epochs): 65.3%
  • Ensemble (Majority Vote): 85.1%
• Analysis: The results indicate that while full fine-tuning with more epochs (v3) led to overfitting or performance degradation in this specific setup, the LoRA approach and the ensemble voting strategy yielded the best results. The ensemble approach effectively mitigated the weaknesses of individual models.

5. Significance

This work demonstrates that Multimodal Large Language Models, when combined with appropriate temporal segmentation strategies, can significantly outperform traditional methods in detecting complex, subtle emotional states like ambivalence and hesitancy.

• Practical Application: The framework provides a scalable solution for digital health and behavioral intervention systems, enabling automated detection of psychological conflicts that often lead to delayed or abandoned behavioral changes.
• Technical Insight: It highlights the importance of aligning input granularity (clip length) with the specific temporal nature of the target emotion and the computational constraints of large models.

The code for the framework is publicly available at the provided GitHub repository.