BROTHER: Behavioral Recognition Optimized Through Heterogeneous Ensemble Regularization for Ambivalence and Hesitancy

1. Problem Statement

The paper addresses the challenge of recognizing Ambivalence and Hesitancy (A/H) in naturalistic video settings. Unlike basic emotions (e.g., joy, anger) which are often distinct and static, A/H represents a subtle, "gray area" psychological state characterized by inner conflict and temporal shifts.

• Key Challenges:
  • Subtlety: A/H manifests through micro-cues in speech pacing, word choice, and vocal tone rather than overt facial expressions.
  • Multimodal Complexity: Detecting A/H requires synthesizing visual, acoustic, and linguistic data over time.
  • Overfitting Risk: Standard emotion recognition systems often force inputs into fixed categories, failing to capture the contradictions and timing shifts inherent in A/H. Furthermore, complex multimodal pipelines are prone to overfitting on training data, leading to poor generalization on unseen "in-the-wild" data.

2. Methodology

The authors propose a highly regularized, multimodal fusion pipeline named BROTHER. The approach consists of four main stages:

A. Multimodal Feature Extraction

The system extracts features from four distinct modalities using state-of-the-art foundational models:

1. Visual: Uses SigLip2 (a vision-language model) to extract embeddings from face-cropped frames (detected via RetinaFace). Features are filtered for noise (using Median Absolute Deviation) and aggregated via normal, first, and second derivatives to preserve temporal dynamics, resulting in a 2304-dimensional vector reduced to 512 via PCA.
2. Audio: Uses HuBERT to extract unsupervised audio embeddings from 1-second chunks, capturing non-linguistic cues like tone and rhythm.
3. Textual: Uses F2LLM (Fine-tuned for Large Language Models) to embed full video transcripts, capturing authentic colloquialisms and global context.
4. Statistical (The 4th Modality): A novel modality derived mathematically from the first three. It aggregates temporal and structural patterns (min, max, mean, std) to summarize localized hesitancy and broader ambivalence.
   • Visual/Audio Stats: Includes valid chunk ratios, spectral centroids, pitch variance, and silence ratios.
   • Textual Stats: Specifically engineered to detect hesitant language (fillers, hedging, corrections) via sentence-level cosine similarity and ambivalent language via prompt-based embedding strategies targeting neutral, negative, positive, and mixed poles.

B. Heterogeneous Classification Committee

Instead of a single model, the authors train a "committee" of models to handle the 15 possible combinations of the four modalities ($2^4 - 1$).

• Architectures: For each modality combination, three distinct classifiers are trained:
  • Multi-Layer Perceptron (MLP) with Gaussian noise and Dropout.
  • Random Forest (RF) via LightGBM.
  • Gradient Boosted Decision Trees (GBDT) via LightGBM.
• Selection Criteria: Models are evaluated on the validation set using Binary Cross-Entropy (BCE) loss (rather than accuracy) to ensure well-calibrated probability outputs. The model with the lowest BCE loss is selected as the representative for that specific modality combination, resulting in a final ensemble of 15 optimized base models.

C. Particle Swarm Optimization (PSO) Ensemble

To fuse the 15 base models, the authors employ Particle Swarm Optimization (PSO) to determine optimal hard-voting weights.

• Hard Voting: Unlike soft voting (averaging probabilities), each model casts a binary vote based on an optimized threshold. The final decision is the weighted sum of these votes.
• Regularized Fitness Function: To prevent overfitting, the PSO fitness function includes a train-validation gap penalty ($\lambda$):
  $$\text{Fitness} = \frac{2 \cdot F1_{val} \cdot F1_{train}}{F1_{val} + F1_{train}} - (\lambda \cdot |F1_{train} - F1_{val}|)^2$$
  This forces the swarm to find weights that maximize generalization (high validation F1) while minimizing the gap between training and validation performance.

3. Key Contributions

1. Specialized Statistical Modality: The introduction of a fourth modality composed entirely of statistical features derived from raw data, specifically designed to capture temporal variations and behavioral markers (hesitation/ambivalence) that standard deep learning embeddings might miss.
2. Heterogeneous Ensemble Strategy: Moving beyond a single architecture, the paper demonstrates that different modality combinations benefit from different classifier types (e.g., MLPs for simple text, Random Forests for complex multimodal concatenations).
3. PSO-Driven Regularization: The implementation of a PSO-based hard-voting ensemble with a dynamic penalty term ($\lambda$) that actively suppresses redundant or overfitted classifiers, ensuring robustness on unseen data.
4. Context-Aware Feature Engineering: The use of prompt-based embedding strategies and specific linguistic dictionaries to mathematically isolate "hesitant" and "ambivalent" language patterns.

4. Results

The system was evaluated on the BAH dataset from the 10th ABAW competition.

• Unimodal Performance: The Text modality was the strongest independent predictor (Validation Macro F1: 0.7275), confirming that linguistic context is crucial for A/H detection. The Video modality alone performed poorly (F1: 0.4704), highlighting the difficulty of visual-only detection for this state.
• Ensemble Performance:
  • The Random Forest (RF) algorithm was the most successful classifier for complex, high-dimensional multimodal combinations.
  • The PSO Ensemble with a 20% penalty ($\lambda = 0.2$) achieved the best results on the unseen test set.
  • Test Macro F1-score: 0.7465
  • Test Weighted F1-score: 0.7559
  • Validation Macro F1-score: 0.7355
• Regularization Impact: Increasing the penalty $\lambda$ forced the PSO to zero out redundant models. At $\lambda=0.2$, the system achieved peak generalization, whereas lower penalties led to overfitting (high train F1, lower test F1).

5. Significance

The paper demonstrates that treating ambivalence and hesitancy as a multimodal temporal conflict rather than a static emotion yields superior results.

• Generalization: The proposed framework effectively bridges the gap between training and real-world ("in-the-wild") performance by explicitly penalizing overfitting during the ensemble optimization phase.
• Interpretability: The results show that while language is the primary anchor for detecting A/H, the synergistic fusion of visual and acoustic signals, when intelligently weighted by a regularized committee, provides a robust framework for behavioral analysis.
• Practical Application: This approach offers a scalable, computationally efficient alternative to massive generative models for detecting complex psychological states in healthcare and behavioral science applications.