BALM: A Model-Agnostic Framework for Balanced Multimodal Learning under Imbalanced Missing Rates

1. Problem Statement

Multimodal learning systems often suffer from modality imbalance, where information-rich modalities dominate optimization while weaker or partially missing modalities contribute less. This issue is exacerbated in realistic scenarios characterized by Imbalanced Missing Rates (IMR).

• The Gap: Most existing research assumes a Shared Missing Rate (SMR), where all modalities are dropped with the same probability. However, in real-world applications (e.g., sensor failures, recording noise), modalities are often absent with different probabilities (e.g., audio might be missing 10% of the time while text is missing 70%).
• The Consequence: Under IMR, two coupled challenges arise:
  1. Representation Imbalance: Heterogeneous missing patterns distort unimodal feature distributions, hindering consistent cross-modal fusion.
  2. Learning Imbalance: Gradients are dominated by frequently observed modalities. Modalities with high missing rates receive fewer samples and weaker supervision, leading to slow convergence and biased optimization where dominant modalities overfit while others underfit.

2. Methodology: The BALM Framework

The authors propose BALM (Balanced Agnostic Learning under Imbalanced Missing Rates), a lightweight, model-agnostic plug-in framework designed to be integrated into existing multimodal backbones (specifically Multimodal Emotion Recognition models) without architectural changes. BALM operates at two levels:

A. Feature Calibration Module (FCM) - Representation Level

The FCM addresses the distortion of unimodal features caused by uneven exposure. Instead of imputing missing signals, it recalibrates available features using global context.

• Global Descriptor Aggregation: It computes a global descriptor for each modality by averaging available features across the batch.
• Cross-Modal Context: These descriptors are concatenated and passed through a fully connected layer to capture shared cross-modal context.
• Adaptive Recalibration: The global context is projected into modality-specific calibration weights. These weights are applied via a gating mechanism (sigmoid activation) to rescale the unimodal features.
• Goal: This aligns feature distributions across varying missing patterns, establishing a shared representation basis and correcting representation bias.

B. Gradient Rebalancing Module (GRM) - Optimization Level

The GRM addresses the optimization bias where dominant modalities drive gradient updates. It modulates gradient magnitudes and directions using two complementary strategies:

1. Distribution-Driven Modulation:

   • Lightweight unimodal prediction heads are added to estimate the learning progress of each modality independently.
   • The module calculates the Kullback-Leibler (KL) divergence between the unimodal prediction and the main multimodal prediction.
   • A modulation coefficient ($\mu^m$) is derived based on the change in this divergence. Modalities that are learning faster (lower divergence change) have their gradients attenuated, while slower modalities are amplified.

2. Spatial-Driven Modulation:

   • This strategy reorients gradients toward a balanced multimodal direction.
   • It measures the cosine similarity between the gradient of a specific modality's prediction head and the gradient of the main multimodal classifier.
   • A spatial loss minimizes the weighted absolute deviation of these similarities, ensuring that modality-specific gradients do not diverge too far from the global optimization objective.

Overall Objective: The final loss combines the task loss (e.g., Cross-Entropy) with the modulation loss, weighted by a hyperparameter $\tau$.

3. Key Contributions

1. Problem Formulation: The paper formally defines and analyzes the Imbalanced Missing Rate (IMR) scenario, demonstrating that traditional SMR assumptions fail to capture the instability and bias present in realistic multimodal settings.
2. Model-Agnostic Framework: BALM is a plug-in solution that requires no redesign of existing backbones. It integrates seamlessly with diverse architectures (e.g., GCNet, MMGCN, MMDFN).
3. Dual-Level Rebalancing: The novel combination of Feature Calibration (FCM) for representation alignment and Gradient Rebalancing (GRM) for optimization dynamics provides a comprehensive solution to both distributional and gradient-level imbalances.
4. Extensive Empirical Validation: The framework is validated on multiple benchmarks (IEMOCAP and CMU-MOSEI) across various missing configurations, showing consistent improvements over state-of-the-art missing-modality and imbalance-aware models.

4. Experimental Results

The authors evaluated BALM on IEMOCAP and CMU-MOSEI datasets under various IMR configurations (e.g., missing rates of 0.3, 0.5, 0.7 assigned differently to Audio, Language, and Visual modalities).

• Performance Gains:
  • BALM consistently improved Accuracy (Acc) and Weighted F1-score (w-F1) across all tested backbones.
  • On IEMOCAP, integrating BALM with GCNet yielded up to +3.27% accuracy gain over the original model.
  • MMDFN+BALM achieved significant improvements, outperforming the mean of Group 1 (specialized missing-modality models) by up to +7.76% in w-F1 under specific IMR settings.
• Robustness:
  • BALM maintained superior performance even under severe missing conditions (e.g., 90% language missing), where baseline models degraded significantly.
  • It also demonstrated robustness under Shared Missing Rate (SMR) settings, indicating that the framework generalizes well even when missing rates are balanced.
• Ablation Studies:
  • Removing FCM or GRM individually resulted in performance drops, confirming that both representation calibration and gradient rebalancing are necessary.
  • The study highlighted that lexical (text) modalities are the most discriminative yet most vulnerable to high missing rates; BALM was particularly effective in mitigating the performance drop when text was heavily missing.

5. Significance

This paper makes a critical contribution to the field of multimodal learning by shifting the focus from idealized Shared Missing Rate assumptions to realistic Imbalanced Missing Rate scenarios.

• Practical Applicability: By being model-agnostic and plug-in, BALM can be easily adopted by practitioners to improve the robustness of existing multimodal systems in real-world deployments where data quality and sensor availability vary.
• Theoretical Insight: It provides a formal analysis of how missing rates scale gradient contributions, offering a theoretical basis for why standard models fail under IMR and how adaptive gradient modulation can restore equilibrium.
• Future Direction: The work sets a new standard for evaluating multimodal models, suggesting that future benchmarks must account for heterogeneous missing patterns rather than uniform dropout.

In summary, BALM effectively decouples the issues of feature distribution distortion and optimization bias, offering a robust, generalizable solution for multimodal learning in imperfect, real-world data environments.