CALM-AH: An ABAW11-Calibrated Multimodal Ensemble with Reliability-Gated Multi-Expert Consensus for Video-Level Ambivalence and Hesitancy Recognition

Technical Summary: CALM-AH and RG-MEC for Video-Level Ambivalence and Hesitancy Recognition

Problem Definition
The paper addresses the challenge of recognizing Ambivalence and Hesitancy (A/H) in naturalistic interview videos, as defined by the ABAW11 Video Recognition Challenge. Unlike standard emotion recognition, A/H represents subtle behavioral states characterized by conflicting evaluations, delayed responses, self-correction, or difficulty committing to an action. These states are often expressed through a complex interplay of linguistic uncertainty, acoustic pauses, prosodic variations, and visual discomfort, which may be temporally distributed and cross-modal. The task requires assigning a binary label (A/H present vs. absent) to entire video clips rather than individual frames. The evaluation is conducted on participant-disjoint data partitions, meaning the system must generalize to unseen individuals, a constraint that complicates the use of high-capacity models prone to overfitting. The primary performance metric is Macro-F1, ensuring equal importance is placed on recognizing both A/H and No-A/H samples.

Methodology
The proposed solution consists of two main components: the CALM-AH ensemble and the Reliability-Gated Multi-Expert Consensus (RG-MEC) mechanism.

1. CALM-AH (Calibrated Multimodal Ensemble):

   • Feature Extraction: The system extracts four complementary representations:
     • Textual: Encoded via F2LLM to capture semantic cues of uncertainty and self-correction.
     • Acoustic: Encoded via HuBERT to capture non-lexical information like pauses and prosody.
     • Visual: Faces are detected (RetinaFace) and encoded frame-wise via SigLIP2, aggregated with temporal differences, and reduced via PCA.
     • Behavioral-Statistical: A 102-dimensional vector derived from visual consistency, acoustic activity, silence patterns, and linguistic indicators.
   • Ensemble Construction: The authors construct 15 non-empty combinations of these feature branches. For each combination, they train three candidate classifier families (MLP, LightGBM Random Forest, and LightGBM GBDT).
   • Calibration: Model selection is based on the lowest validation binary cross-entropy. Subsequently, the decision threshold for each selected model is independently optimized to maximize validation Macro-F1.
   • Fusion: The 15 calibrated binary predictions are fused using fixed-weight hard voting, utilizing weights transferred from the BROTHER system to provide regularization under participant-disjoint evaluation.

2. RG-MEC (Reliability-Gated Multi-Expert Consensus):

   • Architecture: This is an asymmetric, decision-level ensemble designed to correct the initial prediction without introducing instability. It comprises:
     • Anchor: An initial prediction system (providing stability).
     • Correction Committee: Three complementary experts: CALM-AH (task-calibrated multimodal evidence), AffectGPT (multimodal-affective reasoning), and a GPT-based semantic verifier (evaluating semantic/pragmatic evidence of uncertainty).
   • Logic: The system operates on a unanimity-gated principle. The initial anchor prediction is retained by default. It is overridden only if all three correction experts unanimously agree on the same alternative class (either all predicting A/H or all predicting No-A/H).
   • Constraint: No single expert or pair of experts can override the anchor. This design limits the propagation of isolated errors while permitting bidirectional correction when task-specific, affective, and semantic evidence are fully consistent.

Key Contributions

1. CALM-AH: A calibrated heterogeneous ensemble that integrates textual, acoustic, visual, and behavioral-statistical features for A/H recognition, systematically optimizing classifier selection and decision thresholds for the specific ABAW11 data partitions.
2. Fusion Analysis: A systematic comparison demonstrating that independently calibrated hard decisions fused via fixed-weight voting provide more reliable generalization than continuous-margin, question-conditioned, or regularized linear fusion strategies in this context.
3. RG-MEC: An asymmetric inference-time consensus mechanism that assigns non-exchangeable roles to decision sources. It introduces a "stability anchor" protected by a "unanimity-gated correction committee," explicitly allocating decision authority based on source reliability rather than simple majority voting.

Results
Evaluated on the participant-disjoint public test set of the ABAW11 dataset:

• The CALM-AH system alone achieved a Macro-F1 of 0.7525, outperforming the initial anchor (0.7485) and showing balanced performance across both classes (A/H F1: 0.8275; No-A/H F1: 0.6776).
• The complete RG-MEC system achieved a Macro-F1 of 0.7771, with improved class-wise metrics (A/H F1: 0.8455; No-A/H F1: 0.7088) and an overall accuracy of 0.7981.
• The ablation study confirmed that the full RG-MEC rule outperforms intermediate baselines, such as a simple intersection of the anchor and CALM-AH.

Significance and Claims
The paper claims that the primary contribution of RG-MEC is not merely the addition of more predictors, but the explicit allocation of source-dependent decision authority. By treating the initial system as a stability anchor and requiring unanimous agreement from a diverse set of correction experts to override it, the method balances the need for correction with the need to prevent error propagation from isolated experts. The authors note that the RG-MEC rule was developed as a post-hoc, challenge-specific refinement informed by earlier trial submissions, and thus is reported as a specific decision heuristic for the competition context rather than a general validation-only model selection result. The work underscores the importance of combining linguistic, acoustic, and visual information without treating every cross-modal disagreement as evidence of ambivalence, particularly under the constraints of limited participant-level supervision.