📊 statistics

Multimodal Deep Generative Model for Semi-Supervised Learning under Class Imbalance

This paper proposes a multimodal deep generative model for semi-supervised learning under class imbalance that leverages shared latent variables, Student's t-distributions to capture heavy-tailed data characteristics, and a $\gamma$ -power divergence objective to outperform existing methods on partially labeled, imbalanced multimodal datasets.

Original authors: Heegeon Yoon, Heeyoung Kim

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Heegeon Yoon, Heeyoung Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize different types of fruit. You have a huge pile of photos, but there's a catch: you have thousands of pictures of apples, but only a handful of pictures of rare, exotic berries.

If you just show the robot the whole pile, it will get really good at spotting apples but will likely think every rare berry is just a weird-looking apple. This is the problem of class imbalance.

Now, imagine the robot doesn't just see photos; it also hears descriptions of the fruit and feels their texture (if it had sensors). This is multimodal data (using different types of information at once).

Finally, imagine you only have labels (names) for the apples, but the rare berries are all unlabeled. You have to guess what the berries are based on the few apples you know. This is semi-supervised learning.

This paper introduces a new AI model called SSMVAE-CI designed to solve all three of these problems at once: handling the "rare fruit" problem, using multiple senses (sight, sound, text), and learning from mostly unlabeled data.

Here is how it works, broken down with simple analogies:

1. The "Specialist Team" Approach (Multimodal Encoders)

Most AI models try to mash all the data (photos, text, audio) into one giant blender right at the start. The authors say, "No, let's keep the specialists separate."

The Analogy: Imagine a detective team. One detective is an expert in photos, another in audio, and another in text. They don't mix their clues immediately. Instead, they each write a report on what they see, and then they meet in a "central room" (the latent space) to compare notes.
The Benefit: This allows the model to understand that a picture of a berry and a text description of a berry are related, even if they look very different. The model uses a "Product-of-Experts" method, which is like having the team vote on the final conclusion based on their individual reports, rather than forcing them to agree before they even look at the data.

2. The "Heavy-Tailed" Safety Net (Student's t-Distribution)

Standard AI models usually assume that data is distributed like a perfect bell curve (Gaussian distribution). In a bell curve, the "rare" stuff (the tails) is so thin that the model often ignores it or tries to force it to look like the common stuff.

The Analogy: Imagine a bell curve is like a tightrope. If you are a rare berry, you are way off the tightrope in the deep grass. A standard model tries to pull you back onto the tightrope, making you look like an apple.
The Solution: This paper swaps the tightrope for a wide, trampoline-like net (the Student's t-distribution). This net has "heavy tails," meaning it has plenty of room in the deep grass. It allows the rare berries to stay where they naturally belong without being forced to look like apples. This prevents the model from "over-regularizing" (forcing) the rare data into the common patterns.

3. The "Smart Guessing" Game (Gamma-Power Divergence)

The model needs to learn from both the labeled apples and the unlabeled berries. To do this, it uses a special mathematical tool called $\gamma$ -power divergence.

The Analogy: Think of this as a flexible ruler. A standard ruler (like KL divergence) is rigid; if the data doesn't fit perfectly, it breaks or gets confused. The $\gamma$ -power ruler is stretchy and forgiving. It allows the model to say, "I know this rare berry doesn't look exactly like the apples I've seen, but it's close enough to be a berry, and I won't punish it too harshly for being different."
The Result: This helps the model learn effectively even when the data is messy, imbalanced, or missing pieces.

4. What Happens When Data is Missing?

In the real world, sometimes you have a photo but no audio, or text but no image.

The Analogy: If a detective team loses one member (e.g., the audio expert is sick), a rigid team might stop working. But because this model uses the "wide net" (t-distribution) and the "specialist" approach, it can still make a good guess using just the photo and text. It treats the missing data as a "soft" signal rather than a hard stop, allowing it to learn from incomplete samples without throwing them away.

The Results

The authors tested this model on real-world scenarios:

Handwritten digits vs. House numbers: A mix of two image types.
Food images and recipes: A mix of photos and text.
Video, audio, and text: A mix of three types of data from people talking.

In every test, especially when the "rare" classes were very scarce and the data was missing labels, this new model outperformed existing methods. It was particularly good at correctly identifying the "rare fruits" (minority classes) that other models kept misidentifying as "apples" (majority classes).

In short: This paper built a smarter, more flexible AI that uses a team of specialists, a wide safety net for rare data, and a forgiving ruler to learn from messy, incomplete, and unbalanced information better than previous methods.

Technical Summary: Multimodal Deep Generative Model for Semi-Supervised Learning under Class Imbalance

Problem Statement
Deep neural networks (DNNs) often struggle with class-imbalanced data, exhibiting bias toward majority classes and poor generalization on minority classes. This issue is exacerbated in semi-supervised learning (SSL) settings where pseudo-labels for unlabeled data are generated based on imbalanced labeled sets, propagating and reinforcing the bias. Furthermore, real-world data is frequently multimodal (combining diverse sources like images and text), yet existing solutions typically address class imbalance, multimodality, or partial supervision in isolation. Few methods tackle the intersection of all three challenges, and those that do often rely on zero-shot foundation models that lack domain specificity or require substantial computational resources. The core problem addressed is the lack of a unified framework capable of robustly training on partially labeled, multimodal data with imbalanced class distributions.

Methodology: SSMVAE-CI
The authors propose the Semi-Supervised Multimodal Variational Autoencoder for Class-Imbalanced Learning (SSMVAE-CI). This is a unified generative framework that integrates three key technical components:

Multimodal Architecture with Product-of-Experts (PoE):
The model extends the semi-supervised VAE framework to handle $M$ modalities ( $x_1, ..., x_M$ ). It employs modality-specific encoders that share a common latent space containing both a continuous latent variable $z$ and a discrete class variable $y$ . To manage the complexity of inferring the joint posterior $p(y, z | x_1, ..., x_M)$ without requiring an exponential number of inference networks, the model utilizes a Product-of-Experts (PoE) framework. This approximates the joint posterior as the product of individual modality posteriors conditioned on a prior expert, mathematically formulated as:
$q_\phi(y, z|x_1, ..., x_M) = q_\phi(y|x_1, ..., x_M)p(z) \prod_{m=1}^M q_\phi(z|x_m, y)$
This allows for scalable inference across multiple modalities.
Heavy-Tailed Distributions (Student's t):
To address class imbalance, the model replaces the standard Gaussian distributions used in typical VAEs with Student's t-distributions for the prior, encoder, and decoder. The heavy-tailed nature of the t-distribution allows the model to maintain non-negligible probability density in sparse regions of the latent space where minority-class samples reside. This prevents the over-regularization of minority samples that often occurs with Gaussian priors, thereby preserving semantically meaningful but underrepresented patterns.
$\gamma$ -Power Divergence Objective:
The authors derive a new objective function based on $\gamma$ -power divergence to train the model on both labeled and unlabeled data.
- For labeled data, the objective extends the $\gamma$ -loss (used in t3VAE) to include the observed class variable $y$ .
- For unlabeled data, the class variable $y$ is treated as latent. The objective marginalizes $y$ through the $\gamma$ -divergence, deriving a tractable lower bound analogous to the Evidence Lower Bound (ELBO).
- The overall objective combines the generative loss with a discriminative classification loss weighted by a hyperparameter $\alpha$ , balancing coherent latent representation with classification performance.

Key Contributions
The paper claims five primary contributions:

Unified Framework: It is the first generative model to jointly tackle semi-supervision, multimodality, and class imbalance within a single probabilistic framework, reconciling the differing modeling requirements of these challenges.
PoE Inference Mechanism: It develops a PoE inference mechanism that supports both continuous and discrete latent variables across multiple modalities, enabling flexible and scalable modeling.
Generalized Heavy-Tailed Modeling: It generalizes the use of Student's t-distributions to a multimodal, semi-supervised setting, enhancing robustness to class imbalance through heavy-tailed posterior approximations.
Novel Objective Function: It derives a new training objective based on $\gamma$ -power divergence, ensuring stable and expressive learning from both labeled and unlabeled data under imbalanced distributions.
Empirical Validation: Extensive experiments demonstrate consistent outperformance over baseline methods in semi-supervised, multimodal, and class-imbalanced settings.

Experimental Results
The model was evaluated on three datasets:

MNIST-SVHN: A synthetic multimodal dataset combining handwritten digits and house numbers.
UPMC Food-101: A real-world dataset with food images and text descriptions.
CMU-MOSEI: A complex dataset with video, audio, and text modalities for sentiment analysis.

Performance Highlights:

Superior Accuracy: SSMVAE-CI consistently outperformed baselines (including composite methods combining SSL, imbalance handling, and fusion strategies like CReST, Adsh, and LDAM) in both overall accuracy and minority-class accuracy.
Robustness to Imbalance: Performance gains were most pronounced at high imbalance ratios ( $r=100$ ) and low labeled data proportions ( $\beta=10\%$ ). For instance, on MNIST-SVHN with $r=100$ and $\beta=10\%$ , the model achieved 74.79% overall accuracy and 59.07% minority accuracy, significantly outperforming the next best baseline.
Handling Missing Modalities: In experiments where one modality was missing for half the training data, SSMVAE-CI trained on the full dataset (including incomplete samples) showed the smallest performance drop (2.08% overall accuracy) compared to baselines or models trained only on complete data. This demonstrates the model's ability to utilize incomplete samples as training signals.
Ablation Studies: Removing the Student's t-distribution (reverting to Gaussian) or the PoE structure (reverting to simple fusion) resulted in significant performance degradation, confirming the necessity of both components for handling class imbalance and multimodal correlations.

Significance and Claims
The paper posits that SSMVAE-CI offers a structured, data-efficient approach for domains where data is naturally multimodal, partially labeled, and imbalanced. The authors highlight potential applications in medical diagnosis (e.g., combining X-rays with patient history where disease cases are rare) and financial risk assessment (e.g., detecting fraud in multimodal transaction data).

The authors maintain a modest theoretical stance, acknowledging that while the model provides extensive empirical validation, formal convergence or generalization guarantees are not derived due to the heterogeneity of the probabilistic components. However, they assert that the unified probabilistic formulation ensures theoretical consistency for semi-supervised multimodal learning while maintaining robustness to class imbalance, addressing a gap that neither MVAE nor t3VAE could fill individually. Future work is proposed to address distribution mismatches between labeled and unlabeled sets.