🤖 machine learning

Beyond Modality Fusion: Deep Ensembles for Multimodal Classification

This paper demonstrates that deep ensembles of unimodal networks can outperform state-of-the-art late-fusion and intermediate-fusion multimodal classification methods, particularly under modality imbalance, by proposing a scalable heuristic for model allocation and validating these findings across synthetic and real-world datasets.

Original authors: Ilya Burenko, Dmitry Vetrov

Published 2026-07-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ilya Burenko, Dmitry Vetrov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a difficult puzzle. You have two different teams of experts helping you: one team is great at looking at pictures (visual), and the other is great at listening to sounds (audio).

For a long time, the standard way to solve this puzzle was to build a single "Super-Brain" that tried to learn from both teams at the same time. This Super-Brain would take the notes from the picture team and the sound team, glue them together, and try to figure out the answer.

However, the authors of this paper discovered a problem with this Super-Brain approach. If the picture team is very experienced and the sound team is still learning, the Super-Brain tends to ignore the sound team entirely. It gets so distracted by the easy answers from the picture team that it stops listening to the sound team, leading to a worse overall score than if it had just listened to the picture team alone.

The New Idea: A "Council of Experts"

Instead of building one giant Super-Brain, the authors propose a different strategy: The Council of Experts.

Imagine you don't build one giant brain. Instead, you hire a group of smaller, specialized brains.

You hire several "Picture Brains" that only look at images.
You hire several "Sound Brains" that only listen to audio.
You train each of them independently. They never talk to each other while learning; they just focus on their own specific job.

When it's time to solve the puzzle, you ask every single brain in the council for their opinion. Then, you take the average of all their answers to make the final decision.

What Did They Find?

The paper ran many experiments (like a giant science fair) to see which method works better: the "Super-Brain" (Late Fusion) or the "Council of Experts" (Deep Ensembles).

The Council Wins: In almost every test, the Council of Experts beat the Super-Brain. Even when the sound team was much weaker than the picture team, the Council still performed better.
The "Too Many Cooks" Problem: The Super-Brain often gets confused when one team is much stronger than the other. It tries to please everyone but ends up pleasing no one. The Council avoids this because each expert is allowed to be the best at their own specific job without being forced to compromise during training.
Scaling Up:
- Small Council: If you only have a tiny council (e.g., 2 experts), it's actually better to hire two experts from the stronger team (e.g., two Picture Brains) rather than mixing them with the weaker team.
- Big Council: As you hire more experts (scaling up), it becomes beneficial to add experts from the weaker team. The "weaker" experts add just enough extra information to push the average score higher, but only when the council is large enough to handle them.
A Simple Rule of Thumb: The authors created a simple math formula (a heuristic) to tell you exactly how many experts to hire from each team. You don't need to guess or run expensive tests; just look at how well each team performs individually, and the formula tells you the perfect mix.

The "Synthetic Playground"

To prove this wasn't just a fluke, the authors built a "synthetic playground." Imagine a video game where they can artificially control how good the picture team is versus the sound team. They could make the sound team terrible, or make them equal.

They tested their "Council" strategy in this game.
They tested it on real-world data (like recognizing emotions in videos or detecting sarcasm in text).
Result: The Council worked great in both the game and the real world.

The Bottom Line

The paper concludes that we don't always need to force different types of data (like text and images) to merge into a single complex model. Sometimes, the best approach is to keep them separate, train many independent models on each type of data, and simply let them vote on the final answer.

This "voting" method is not only more accurate but also easier to manage when one type of data is much harder to learn than the other. It turns out that a group of specialized, independent thinkers often solves problems better than one giant, overworked generalist.

Technical Summary: Beyond Modality Fusion: Deep Ensembles for Multimodal Classification

Problem Statement

In multimodal classification, the standard approach involves "late-fusion" networks, which concatenate features extracted by unimodal neural networks and pass them to a joint classifier. However, when significant modality imbalance exists (where one modality is significantly more predictive or converges faster than another), naive late-fusion networks often underperform the best single unimodal baseline. This phenomenon occurs because the optimization process tends to under-explore the slower-converging (weaker) modality, leading to suboptimal feature extraction. While various regularization techniques (e.g., gradient modulation, loss regularization) have been proposed to balance learning, the optimal method for combining unimodal features in late-fusion settings remains an open question.

Methodology

The authors propose an alternative to explicit modality fusion: Heterogeneous Deep Ensembles (HeDEs). Instead of training a single network to fuse features, HeDEs consist of multiple unimodal networks trained independently on different modalities. The final prediction is obtained by averaging the output logits of these independent models.

Key methodological components include:

Ensemble Construction: For a bimodal dataset, the authors train $n_1$ classifiers on modality 1 and $n_2$ on modality 2. The ensemble prediction is the average of all $N_{ens} = n_1 + n_2$ logits. When $n_2=0$ , this reduces to a Homogeneous Deep Ensemble (HoDE).
Heuristic for Model Allocation: To avoid computationally expensive exhaustive searches for the optimal number of models per modality ( $n_1, n_2$ ), the authors propose a heuristic based on the validation loss ( $l_i$ ) of unimodal classifiers. The number of models for modality $i$ is proportional to $1/l_i$ :
$n_i = \text{round}\left( N_{ens} \frac{1/l_i}{\sum_j 1/l_j} \right)$
This heuristic predicts that under extreme imbalance, homogeneous ensembles of the stronger modality are preferable at small scales, while heterogeneous ensembles become optimal as the ensemble size grows.
Synthetic Dataset Framework: To systematically study modality imbalance, the authors introduce a synthetic construction method. Starting from a unimodal dataset, they split it into non-overlapping subsets to create bimodal (or trimodal) pairs. By varying the size of these subsets, they control the predictive strength and convergence speed of each modality, allowing for the training of overparameterized networks to observe optimization challenges not present in prior synthetic benchmarks.
Scaling Laws: The authors fit power laws to the test loss of ensembles as a function of size ( $N$ ) and proportion ( $\theta$ ), demonstrating that the asymptotic performance of the ensemble can be predicted from the losses of individual unimodal models.

Key Contributions

Superiority of Deep Ensembles: The paper demonstrates that deep ensembles of unimodal networks consistently outperform state-of-the-art late-fusion methods (including those designed to handle imbalance like OGM-GE, MMPareto, and InfoReg), intermediate-fusion techniques (MMTM, cross-attention), and hybrid approaches (I2M2). This holds true even when comparing models with equal parameter counts.
Validation of the Allocation Heuristic: The proposed heuristic for selecting the number of models per modality is empirically validated across synthetic and real-world datasets. It correctly predicts the transition from homogeneous to heterogeneous ensembles as the ensemble size scales up, particularly under conditions of extreme modality imbalance.
Synthetic Multimodal Framework: The authors provide a synthetic dataset construction that allows independent control over the number of modalities and their predictive strength. This framework reveals optimization challenges (such as the faster convergence of weaker modalities in late-fusion) observed in practice, which previous synthetic benchmarks failed to reproduce.
Scaling Laws for Ensembles: The study establishes scaling laws showing that the asymptotic test loss of a heterogeneous deep ensemble is reliably predictable from the unimodal model losses, suggesting that unimodal validation metrics are sufficient to estimate ensemble performance without training the full ensemble.

Experimental Results

The authors evaluated their approach on:

Synthetic Data: Bimodal and trimodal splits of CIFAR-100 and ImageNet-1K.
Real-World Data: CREMA-D (audio-visual), Sarcasm (text-visual), and Kinetics-400 (audio-visual).

Key Findings:

Performance: HeDEs outperformed all late-fusion baselines, intermediate-fusion networks, and hybrid models across all datasets and imbalance levels.
Imbalance Dynamics: Under extreme imbalance and small ensemble sizes, HoDEs (using only the stronger modality) outperformed HeDEs. However, as the ensemble size increased, HeDEs consistently surpassed HoDEs and late-fusion methods.
Heuristic Accuracy: The proposed heuristic accurately predicted the optimal modality allocation in nearly all tested configurations, matching brute-force search results with minimal error.
Calibration: Logit averaging in HeDEs yielded well-calibrated predictions without post-hoc calibration, often outperforming probability averaging.
Cost: While training HeDEs requires training multiple independent models, the inference cost is comparable to late-fusion networks of equal parameter count. The training process is fully parallelizable.

Significance and Claims

The paper claims that explicit modality fusion is not strictly necessary for effective multimodal classification. Instead, the benefits of multimodal data can be captured by ensembles of independently trained unimodal models. The authors argue that their approach offers a robust, parameter-efficient alternative to complex fusion architectures, particularly in scenarios with modality imbalance.

The work suggests that the "greedy" nature of late-fusion optimization, which often leads to the neglect of weaker modalities, can be circumvented by the independent training and aggregation strategy of deep ensembles. Furthermore, the ability to predict ensemble performance and optimal composition using only unimodal validation losses offers a practical tool for resource-constrained multimodal learning.

The authors maintain a modest stance, acknowledging that while their findings are consistent across diverse datasets, the theoretical basis for the allocation heuristic remains unclear, and further evaluation across a broader range of fusion techniques and large-scale multimodal models is needed. They also note that their synthetic construction is class-aligned rather than instance-aligned, though they argue that the phenomena studied (generalization and convergence rates) are preserved in this setting.