📊 statistics

Simultaneous Long-tailed Recognition and Multi-modal Fusion for Highly Imbalanced Multi-modal Data

This paper proposes a novel multi-modal framework for long-tailed recognition that dynamically fuses heterogeneous data sources using confidence-guided weights to overcome class imbalance and outperform existing single-modal methods.

Original authors: Heegeon Yoon, Heeyoung Kim

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Heegeon Yoon, Heeyoung Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive talent show where you need to pick the best performers from thousands of applicants. However, there's a catch: 99% of the applicants are "average" singers, while only a tiny handful are "genius" opera singers. If you train your judges (your AI model) only on this crowd, they will become experts at spotting average singers but will completely miss the geniuses because they've never seen enough of them. This is the problem of Long-tailed Recognition: AI gets biased toward the common stuff and ignores the rare, important stuff.

Now, imagine these applicants don't just sing; they also bring a resume, a video, and a voice recording. This is Multi-modal Data (different types of information). Usually, AI struggles to combine these different sources, especially when the "genius" singers are so rare.

This paper introduces a new system to solve both problems at once. Here is how it works, using simple analogies:

1. The "Specialized Panel of Judges" (Multi-Expert Framework)

Instead of having one big judge try to learn everything, the authors set up a panel of three specialized judges, each with a different personality:

Judge A (The Long-Tail Expert): Trained to love the rare stuff. They are hyper-aware of the "genius" singers.
Judge B (The Balanced Expert): Trained to treat everyone equally, regardless of how common they are.
Judge C (The Reverse Expert): Trained to focus heavily on the common stuff, just to see how they react.

In previous systems, these judges only looked at one thing (like just the video). This new system teaches all three judges to look at everything (video + resume + audio) at the same time.

2. The "Trust Meter" (Modality-Aware Fusion)

Sometimes, a judge might look at a resume and think, "This looks great," while looking at the video and thinking, "This looks blurry and useless."

The paper adds a special "Trust Meter" (called a confidence-guided weight) to the system.

If the "video" is clear and helpful, the Trust Meter says, "Listen to the video!"
If the "resume" is messy or confusing, the Trust Meter says, "Ignore the resume for this specific person."

This happens dynamically. For one applicant, the video might be the most important clue. For another, the resume might be the key. The system automatically decides which piece of information to trust more for each individual case.

3. The "Training vs. Testing" Trick

Here is where the authors had to get creative because of the type of data they used.

For Image Data (Photos): In the past, to make the judges smarter during the final show, the system would take a photo, make a slightly blurry version, and a slightly bright version, and ask the judges to agree on the answer. This "self-supervised" trick helped them adjust their weights on the fly.
For Tabular Data (Spreadsheets/Resumes): You can't really make a "blurry" version of a spreadsheet or a "brighter" version of a list of ages without breaking the data.

The Solution: The authors changed the rules for the spreadsheet data. Instead of trying to adjust the judges' weights during the final show (which is impossible with spreadsheets), they taught the system to figure out the perfect weights during the training phase using the known answers. Once the training is done, the weights are locked in. It's like the judges rehearsing until they know exactly how much to trust the resume vs. the video, so they don't need to guess during the live show.

The Results

The authors tested this system on two things:

Synthetic Data: Mixing two famous image datasets (MNIST and SVHN) to create a fake long-tailed problem.
Real Medical Data: A real-world dataset for detecting melanoma (skin cancer) from images and patient metadata (age, gender, location).

The Outcome:

The new system was much better at finding the "rare" cases (the minority classes) than previous methods.
It didn't just guess; it learned to trust the right information source for the right person.
In the medical dataset, it significantly improved the ability to detect the rare malignant (cancerous) cases compared to other methods, without losing accuracy on the common benign cases.

In Summary

This paper builds a smarter AI team that:

Uses specialized experts to handle rare and common data equally well.
Uses a dynamic trust meter to decide which type of data (image or text) matters most for each specific case.
Adapts its training strategy so it works perfectly even when you have messy spreadsheets that can't be "augmented" like photos.

The result is a system that is much better at spotting the "needle in the haystack" when that needle comes with a mix of different clues.

Technical Summary: Simultaneous Long-tailed Recognition and Multi-modal Fusion for Highly Imbalanced Multi-modal Data

Problem Statement

Deep neural networks (DNNs) often suffer from performance degradation when trained on class-imbalanced data, exhibiting a bias toward majority classes and failing to learn effective representations for minority classes. While recent advancements have addressed long-tailed recognition, they are predominantly restricted to single-modal inputs. Conversely, multi-modal learning approaches effectively exploit complementary information from heterogeneous sources (e.g., images and tabular data) but rarely account for class imbalance.

In real-world applications, such as medical diagnostics, datasets are frequently both multi-modal and long-tailed. For instance, medical records may contain imaging data alongside demographic and clinical history, yet samples of specific conditions (minority classes) are vastly outnumbered by healthy cases. Existing frameworks fail to address these challenges simultaneously: multi-expert models like Self-supervised Aggregation of Diverse Experts (SADE) are limited to single-modal inputs, while multi-modal fusion methods like Multi-modal Dynamics (MMD) are agnostic to long-tailed distributions. This paper proposes a unified framework to bridge this gap, enabling robust recognition in scenarios where data is both multi-modal and highly imbalanced.

Methodology

The proposed framework integrates distribution-specialized multi-expert learning with modality-aware fusion. The architecture consists of three primary components:

1. Modality-Aware Multi-Expert Backbone

Building on the SADE architecture, the method employs three independent expert networks ( $E_1, E_2, E_3$ ), each specialized for a distinct class distribution:

$E_1$ (Forward): Trained with standard softmax cross-entropy loss ( $L_{ce}$ ) to handle long-tailed distributions.
$E_2$ (Uniform): Trained with balanced softmax loss ( $L_{bal}$ ) to handle uniform distributions.
$E_3$ (Backward): Trained with inverse softmax loss ( $L_{inv}$ ) to handle inversely long-tailed distributions.

Unlike the original SADE, which processes single-modal inputs, each expert in this framework is redesigned to accept multi-modal inputs. The inputs from heterogeneous modalities (e.g., images and tabular data) are processed through modality-specific sub-networks and fused into a unified latent representation within each expert.

2. Modality-Assessment and Weighting Module

To effectively fuse heterogeneous data, the framework incorporates a classifier module inspired by MMD. This module includes modality-specific networks that estimate the True Class Probability (TCP) for each modality.

A confidence network ( $g_m$ ) approximates the TCP for each modality $m$ , outputting a confidence score.
These scores act as dynamic weights, modulating the contribution of each modality's feature representation before fusion.
This mechanism ensures that more informative modalities contribute more strongly to the final decision, conditioned on both the input sample and the class distribution.

3. Training and Testing Procedures

The training and testing strategies differ based on the nature of the modalities to address the feasibility of data augmentation:

Case 1: Fully Image-Based Modalities: The method adopts a test-time training strategy similar to SADE. Two stochastic augmentations are generated for each test sample. The aggregation weights ( $w_1, w_2, w_3$ ) for combining the three experts are optimized in a self-supervised manner by maximizing prediction stability between the two augmented views.
Case 2: Mixed Image and Tabular Modalities: Since generating comparable stochastic augmentations for tabular data is often infeasible and can degrade performance, the test-time training strategy is replaced. Instead, the aggregation weights are optimized exclusively during the training phase using ground-truth labels and a standard softmax cross-entropy loss. This assumes the training and test datasets share the same distribution, a reasonable assumption for data collected from the same environment.

Key Contributions

The paper outlines three primary contributions:

Unified Framework: A novel long-tailed recognition framework that simultaneously handles multi-modal inputs and class imbalance, integrating multi-expert learning with modality-aware aggregation.
Specialized Procedures: The design of tailored training and testing strategies that accommodate diverse combinations of input modalities, specifically addressing the limitations of applying test-time augmentation to non-image data.
Empirical Validation: Extensive evaluations on benchmark datasets (MNIST, SVHN) and a real-world medical dataset (SIIM-ISIC Melanoma Classification Challenge) demonstrating the approach's effectiveness.

Experimental Results

The proposed method was evaluated against several baselines, including MMD, SADE, MMD-LA, SADE-LMF, and M2LC-Net.

MNIST and SVHN (Two Image Modalities): The proposed method consistently outperformed baselines in terms of F1 score, particularly under inversely long-tailed test distributions with high imbalance ratios. For example, with a training imbalance ratio of 50 and an inversely long-tailed test distribution, the proposed method improved the F1 score by 0.43 over MMD-LA and 0.51 over M2LC-Net. It also demonstrated superior generalization, maintaining competitive accuracy while significantly boosting performance on minority classes.
SIIM-ISIC Melanoma Dataset (Image and Tabular Modalities): In this real-world medical setting with an imbalance ratio of approximately 55.72, the proposed method achieved the highest F1 score (0.6012) among all compared methods, outperforming the next best baseline (SADE-LMF, 0.5629) by 0.04. While classification accuracy remained comparable to other top-performing models, the significant gain in F1 score highlights the model's ability to better identify the minority class (malignant lesions).
Aggregation Weights: Analysis of the learned aggregation weights confirmed that the proposed method assigns weights logically consistent with the test data distribution (e.g., higher weight to the forward expert for long-tailed tests), whereas baseline methods like SADE-LMF sometimes assigned counter-intuitive weights.

Significance and Claims

The paper claims that the proposed approach effectively addresses the dual challenges of class imbalance and multi-modal fusion, which have largely been treated as separate problems. By extending the multi-expert framework to multi-modal settings and introducing a confidence-guided weighting mechanism, the method achieves robustness across varying test-time distributions.

The authors emphasize that the proposed method does not sacrifice classification accuracy for minority class performance; rather, it enhances generalization to unseen, non-uniform data distributions. The ability to handle mixed modalities (images and tabular data) without relying on infeasible test-time augmentations is presented as a critical advancement for real-world applications where data heterogeneity and imbalance coexist. The work concludes by noting that future efforts will aim to extend this framework to semi-supervised settings to further address scenarios with limited labeled data.