Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis

1. Problem Statement

Multimodal Sentiment Analysis (MSA) aims to infer human emotions by integrating heterogeneous signals from text, audio, and visual modalities. Despite recent advances, existing methods face two critical challenges:

• Modality Imbalance & Competition: Textual data often dominates the learning process due to its high information density and lower noise levels compared to audio and visual signals. This leads to a "Matthew effect" where dominant modalities (text) accumulate larger gradients, overshadowing weaker modalities (audio/visual), which become marginalized and under-optimized.
• Robustness Issues: Current models struggle significantly when modalities are noisy, incomplete, or missing. The imbalance exacerbates this, as the model relies too heavily on the dominant modality, failing to leverage complementary cues from weaker modalities when the dominant one is corrupted or absent.

2. Methodology: The EBMC Framework

The authors propose EBMC (Enhance-then-Balance Modality Collaboration), a two-stage framework designed to first strengthen weak modalities and then dynamically balance their contributions during fusion.

Stage I: Weak Modality Enhancement

This stage focuses on improving the representation quality of all modalities, specifically targeting the weaker ones.

• Modality Semantic Disentanglement (MSD):
  • Decomposes each modality's representation ($z_m$) into a shared component ($z^c_m$) and a modality-specific component ($z^s_m$).
  • Uses InfoNCE contrastive loss to align shared semantics across modalities.
  • Uses cosine similarity minimization to ensure specific components remain distinct and non-redundant.
  • Includes a unimodal prediction loss to ensure specific components retain predictive power.
• Cross-modal Complementary Enhancement (CCE):
  • Takes the disentangled components of a target modality and enriches them with shared and specific components from other modalities.
  • Uses a lightweight enhancement network to generate enhanced features ($\tilde{z}_m$), effectively "compensating" for weak signals in audio/visual modalities using textual context, and vice versa.

Stage II: Modality Balancing & Robustness

This stage addresses the competition between modalities and ensures robustness against noise/missing data.

• Energy-guided Modality Coordination (EMC):
  • Reinterprets multimodal coordination through Energy-based Models (EBMs).
  • Defines a modality-specific Energy Potential ($E(m)$) incorporating representation magnitude, task loss, and predictive uncertainty.
  • Energy-gap Minimization: Enforces a global equilibrium objective to prevent any single modality from having disproportionately low energy (dominance).
  • Energy-gradient Flow: Implements a negative-feedback mechanism where dominant modalities (low energy) receive suppressive gradients, while weak modalities (high energy) receive amplifying gradients. This implicitly rebalances optimization dynamics without manual weight tuning.
• Instance-aware Modality Trust Distillation (IMTD):
  • Estimates sample-level reliability using probabilistic embeddings from the teacher models.
  • Calculates a confidence score based on predictive variance (lower variance = higher confidence).
  • Uses these scores to adaptively modulate fusion weights during knowledge distillation, down-weighting noisy or unreliable modalities on a per-sample basis.

3. Key Contributions

1. Weak-Modality Enhancement Mechanism: A novel approach combining semantic disentanglement and cross-modal compensation to strengthen under-represented modalities (audio/visual) without sacrificing unimodal predictive capability.
2. First EBM-Inspired Coordination: The introduction of an energy-guided modality coordination mechanism that dynamically rebalances modalities via energy potentials and gradient flow dynamics, effectively mitigating modality competition.
3. Instance-Aware Trust Distillation: A module that estimates sample-level modality reliability to adaptively adjust fusion weights, significantly improving robustness in noisy or missing-modality scenarios.
4. Comprehensive Validation: Extensive experiments demonstrating state-of-the-art (SOTA) performance and superior robustness compared to existing baselines.

4. Experimental Results

The model was evaluated on three standard benchmarks: CMU-MOSI, CMU-MOSEI, and IEMOCAP.

• Performance on Full-Modality Tasks:
  • EBMC achieved SOTA or highly competitive results across all datasets.
  • On CMU-MOSI, it improved Acc-7 by 1.89% over the previous best (Semi-IIN) and F1 by 1.19% over GLoMo.
  • On CMU-MOSEI, it achieved an F1 score of 88.07%, outperforming GLoMo by 1.67%.
  • On IEMOCAP (Emotion Recognition in Conversations), it achieved an average F1 of 86.35%, surpassing the strong baseline DMD by 1.27%.
• Robustness under Missing Modalities:
  • In scenarios with missing modalities (e.g., only audio or only visual available), EBMC consistently outperformed baselines.
  • It showed significantly smaller performance degradation compared to other methods, particularly in single-modality settings where text is missing.
• Robustness under Intra-Modality Noise:
  • Under simulated frame-level feature dropping (intra-modality missingness), EBMC maintained superior performance (e.g., +1.52% F1 over LNLN on CMU-MOSI), validating the effectiveness of the IMTD module.
• Ablation Studies:
  • Removing the EMC module caused the largest performance drop (approx. 2.4–2.8% F1), confirming its critical role in balancing modality contributions.
  • Removing MSD or CCE also significantly degraded performance, proving the necessity of enhancement and disentanglement.

5. Significance

• Theoretical Advancement: The paper bridges the gap between representation learning and optimization dynamics by applying Energy-based Model principles to multimodal fusion, offering a principled alternative to heuristic gradient balancing.
• Practical Robustness: By addressing the "Matthew effect" in multimodal learning, EBMC provides a more reliable solution for real-world applications (e.g., mental health monitoring, human-computer interaction) where data is often noisy, incomplete, or unbalanced.
• Generalizability: The framework's success in both Sentiment Analysis and Emotion Recognition in Conversations (ERC) tasks demonstrates its versatility across different multimodal understanding domains.
• Efficiency: The supplementary material indicates that when features are pre-extracted, the core EBMC modules are lightweight, requiring minimal computational overhead compared to the backbone encoders.

In conclusion, EBMC offers a unified, two-stage solution that not only enhances the expressiveness of weak modalities but also dynamically coordinates their contributions, leading to robust and accurate multimodal sentiment analysis even in imperfect data environments.