⚡ electrical engineering

Mitigating Sample-Level Imbalance via Probabilistic Separation for Adaptive Multimodal Fusion

This paper proposes a novel framework that mitigates sample-level modality imbalance in multimodal learning by utilizing a Modality Gap metric and Gaussian Mixture Model to probabilistically separate samples into balanced and imbalanced subgroups, thereby enabling a dynamic two-stage training process that adaptively reallocates optimization priorities and purifies data for superior performance.

Original authors: Zhiwen Yu, Zhaocheng Liu, Xiaoqing Liu, Huanqiang Zeng, C. L. Philip Chen

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhiwen Yu, Zhaocheng Liu, Xiaoqing Liu, Huanqiang Zeng, C. L. Philip Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Human beings perceive the world through a symphony of senses, where sight, sound, and touch combine to create a richer understanding than any single sense could provide alone. In the field of artificial intelligence, researchers strive to replicate this ability by building systems that can process multiple types of data simultaneously, such as video and audio. This approach, known as multimodal learning, has shown great promise in tasks like recognizing emotions in speech or identifying events in videos. However, a persistent problem has hindered these systems from reaching their full potential. When a computer learns from both sight and sound, one sense often becomes dominant, effectively drowning out the other. If the visual data is clear and the audio is noisy, the system learns to rely almost entirely on the pictures, ignoring the sound. This creates an imbalance where the weaker sense fails to improve, and the combined system performs worse than it should, sometimes even underperforming a system that uses only the stronger sense.

A team of researchers has developed a new method to solve this problem by treating every single example in the training data individually, rather than treating the entire dataset as a uniform block. They discovered that within any collection of video and audio clips, there are two distinct types of examples. Some clips are well-balanced, where the visual and audio information are equally clear and helpful. Others are imbalanced, where one sense is so much stronger than the other that it creates confusion during the learning process. The researchers found that these two groups of examples naturally form separate clusters within the data, much like how a crowd might naturally split into two distinct groups based on height. By identifying which examples belong to which group, the system can learn to handle them differently, ensuring that the weaker sense gets the attention it needs to catch up.

The core of their solution involves a two-stage training process. First, the computer is given a "warm-up" period where it learns the basics of both senses without any special adjustments. During this time, the system records how much the predictions from the visual branch and the audio branch disagree with each other for every single clip. This disagreement is measured as a "modality gap." After this initial phase, the researchers analyzed the distribution of these gaps and confirmed their suspicion: the data naturally separated into a large group of balanced examples and a smaller group of imbalanced ones. To make use of this discovery, they used a statistical tool to map out these two groups and calculate the probability that any new example belongs to the balanced group or the imbalanced group.

With this map in hand, the system enters its second, adaptive training stage. Here, the computer changes its learning strategy based on the probability score of each example. For the examples that are well-balanced, the system focuses on fusing the two senses together to maximize the benefit of their combined information. For the examples that are imbalanced, where one sense is struggling, the system applies a stronger penalty to force the two senses to align more closely. This approach ensures that the computer does not simply ignore the difficult examples but instead works harder to correct the bias in those specific cases. The researchers also introduced a mechanism that gradually reduces the intensity of this correction over time, allowing the system to shift its focus from fixing imbalances to perfecting the final classification task as it learns.

The results of this approach were tested on three different datasets involving speech emotion recognition, event localization, and human action recognition. On the speech emotion dataset, which contained over 7,000 video clips, the new method achieved an accuracy of 80.65 percent, significantly outperforming previous state-of-the-art methods. Similar improvements were seen on the other two datasets, with accuracy rates reaching 70.40 percent and 72.61 percent respectively. Beyond just improving the final score, the researchers observed that the method successfully reduced the performance gap between the audio and visual branches, allowing the weaker sense to improve without sacrificing the strength of the dominant one.

In a further experiment, the researchers demonstrated that the statistical model could be used to filter the data itself. By selecting only the high-quality, balanced examples identified by their model, they were able to fine-tune the system on a much smaller subset of the data. Even with fewer examples, the system performed better than when trained on the full, unfiltered dataset. This suggests that the method not only helps the computer learn more efficiently but also acts as a powerful tool for cleaning up noisy data, ensuring that the system learns from the most reliable examples available. The study concludes that by recognizing and adapting to the natural variations in data quality at the sample level, multimodal systems can overcome the limitations of traditional training methods and achieve a more robust and accurate understanding of the world.

Technical Summary: Mitigating Sample-Level Imbalance via Probabilistic Separation for Adaptive Multimodal Fusion

1. Problem Statement

Multimodal learning aims to integrate information from diverse sources (e.g., audio and visual) to achieve performance superior to unimodal approaches. However, a critical challenge known as modality imbalance often hinders this potential. Due to variations in information density and feature extraction difficulty, different modalities exhibit inconsistent convergence rates during joint training. Consequently, the optimization process becomes dominated by the "strong" modality, causing the "weak" modality to remain underfitted or stagnate.

Existing solutions, such as static modulation or heuristic gradient adjustments, often fail to address sample-level distributional variations. Specifically, they do not effectively distinguish between samples where modalities are naturally balanced and "outlier" samples where the modality gap is exacerbated by low data quality. This oversight leads to gradient conflicts and prevents the model from fully exploiting cross-modal complementary information.

2. Methodology

The authors propose a novel framework that quantitatively diagnoses and dynamically mitigates imbalance at the sample level through a two-stage training process: a Warm-up stage and an Adaptive Training stage.

A. Modality Gap Metric (MGM)

To quantify the degree of imbalance for individual samples, the paper introduces the Modality Gap ( $g_i$ ). This metric is defined from two perspectives:

Confidence-based: The difference in predicted probabilities for the ground-truth class between the dominant and weak modalities ( $g_i = s^a_{i,y_i} - s^v_{i,y_i}$ ).
Information-theoretic: The difference in Kullback–Leibler (KL) divergence between the predicted distribution and a uniform distribution for each modality.
These raw values are standardized via Z-score normalization to create a consistent input space.

B. Probabilistic Separation via GMM

Through modeling analysis, the authors observe that the distribution of the Modality Gap exhibits a distinct bimodal structure:

A primary peak near zero, representing modality-balanced samples.
A secondary tail/peak, representing modality-imbalanced samples (outliers with significant prediction bias).

To exploit this structure, the framework employs a Gaussian Mixture Model (GMM) to explicitly model the distribution of the Modality Gap after the warm-up phase. Using Bayes' theorem, the model computes the posterior probability ( $w_{i,k}$ ) that a specific sample $i$ belongs to either the balanced subgroup ( $k=0$ ) or the imbalanced subgroup ( $k=1$ ). These probabilities serve as dynamic soft weights for subsequent optimization.

C. Two-Stage Adaptive Training

Warm-up Stage: The model is trained using a standard objective that includes both multimodal fusion loss and auxiliary unimodal losses ( $L_{wp} = L_f + L_a + L_v$ ). This phase ensures preliminary convergence and establishes reliable feature representations necessary for calculating the Modality Gap.
Adaptive Training Stage: This core stage utilizes the GMM-derived posterior probabilities to construct a sample-level adaptive loss function ( $L^{adap}_i$ $L_{i}^{a d a p}$ ). The loss dynamically reallocates optimization priorities:
- For Balanced Samples ( $w_{i,0}$ high): The loss imposes stronger penalties on the fusion task to maximize the exploitation of cross-modal complementary information.
- For Imbalanced Samples ( $w_{i,1}$ high): The loss imposes stronger geometric constraints (penalizing the deviation of the modality gap from the balanced mean) to rectify bias and narrow the gap.
- Regularization: A term penalizing large modality gaps is applied to all samples.
- Annealing: An annealing coefficient ( $\lambda_t$ ) is introduced to gradually shift focus from correcting imbalance in early adaptive training to optimizing the primary classification task in later stages.

The GMM fitting and network parameter updates are coordinated via an alternating iterative scheme to ensure the probabilistic model remains consistent with the evolving data distribution.

3. Key Contributions

The paper outlines three primary contributions:

Quantitative Metric and Distribution Modeling: The proposal of a Modality Gap metric to assess sample-level imbalance and the first-time application of a Gaussian Mixture Model (GMM) to model its bimodal distribution, revealing the natural coexistence of balanced and imbalanced subgroups.
Sample-Level Adaptive Optimization: The design of a GMM-guided Adaptive Loss function that dynamically weights optimization based on cross-modal balance. This is implemented within a two-stage training scheme (Warm-up and Adaptive Training) to stabilize the optimization process.
Data Purification Strategy: The demonstration that the GMM-based identification of high-quality balanced subsets serves as an effective data purification strategy. Fine-tuning on these filtered subsets significantly enhances baseline performance, even without the full adaptive loss framework.

4. Experimental Results

The proposed method was evaluated on three benchmark datasets: CREMA-D (emotion recognition), AVE (event localization), and KineticSound (action recognition).

State-of-the-Art Performance: The method achieved new SOTA results, outperforming baselines such as OGM-GE, Greedy, and PMR.
- CREMA-D: 80.65% accuracy (vs. 79.70% for the next best, MLA).
- AVE: 70.40% accuracy.
- KineticSound: 72.61% accuracy.
Ablation Studies:
- Removing any component of the adaptive loss (fusion penalty, gap regularization, or GMM weights) resulted in performance degradation.
- Optimizer Strategy: Resetting the optimizer state between the warm-up and adaptive phases was found critical; inheriting momentum from the warm-up phase disrupted the second phase.
- Distribution Sensitivity: While GMM performed best, comparisons with Student's t-Mixture and Laplace Mixture models confirmed the robustness of the Gaussian assumption for this specific task.
Data Purification: Fine-tuning models exclusively on high-quality subsets (selected via GMM posterior probability thresholds) yielded substantial gains over training on the full dataset, validating the "data purification" capability of the method.
Unimodal Evolution: Analysis of unimodal accuracy curves showed that the adaptive training phase successfully bridged the performance gap between modalities, preventing the stagnation of the weaker modality observed in baseline methods.

5. Significance and Claims

The paper claims that its significance lies in shifting the paradigm from static, batch-level imbalance handling to dynamic, sample-level probabilistic separation. By quantitatively diagnosing the bimodal nature of modality gaps, the framework avoids the "one-size-fits-all" approach of existing methods.

The authors assert that their approach effectively resolves the optimization trade-off between balanced and imbalanced samples, allowing the model to simultaneously correct bias in difficult samples and maximize fusion benefits in balanced ones. Furthermore, the ability to use the GMM for data purification offers a practical tool for improving baseline performance without complex architectural changes, highlighting the dual utility of the proposed probabilistic modeling.