🤖 machine learning

Unsupervised Learning for Missing Modalities in Multimodal Learning

This paper introduces UL4M4, a flexible unsupervised framework that imputes missing feature embeddings in multimodal learning using modality-specific normalization and partial-modality distance metrics to achieve robust, state-of-the-art performance even when over 50% of modalities are missing.

Original authors: Hassan Ismkhan, Hamid Bouchahcia

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hassan Ismkhan, Hamid Bouchahcia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex puzzle, like guessing the genre of a movie. Usually, you have all the pieces: the movie poster (visual), the plot summary (text), and maybe even the trailer audio. But in the real world, pieces often go missing. Maybe the poster is lost, the audio file is corrupted, or the text is too short.

Traditional AI models are like rigid puzzle solvers: if a piece is missing, they often give up or make a terrible guess. This paper introduces a new, flexible method called UL4M4 (Unsupervised Learning for Missing Modalities) that acts like a clever detective who can fill in the missing puzzle pieces before trying to solve the final mystery.

Here is how it works, broken down into simple steps:

1. The Problem: The "Missing Piece" Dilemma

In the real world, data is messy. Sensors fail, privacy rules hide information, or files get lost. Most AI models assume they will always get the full picture (text + image + audio). When they don't, their performance crashes. Existing solutions try to "rebuild" the missing piece using complex math, but they are often too specific to one type of task or require the AI to be retrained every time the missing pieces change.

2. The Solution: The "Grouping and Guessing" Strategy

The authors propose a two-step process that is task-independent. This means the "filling in" part doesn't care what the final goal is (whether it's guessing a movie genre or analyzing a mood); it just focuses on making the data complete.

Step 1: The "Group Hug" (Clustering)
Imagine you have a huge room full of people (your data), but some are missing their left shoes, some their right, and some have both.

Instead of trying to match everyone perfectly, the AI groups people into clusters based on the shoes they do have.
It uses a special "distance rule" that says, "It doesn't matter if Person A has 3 shoes and Person B has 1; we can still compare them fairly."
Once grouped, the AI creates a "Group Avatar" for each cluster. This avatar represents the average look of everyone in that group, including the shoes they are missing.

Step 2: The "Greedy Filler" (Imputation)
Now, imagine you have a specific person (a data sample) who is missing their audio shoe.

The AI looks at all the "Group Avatars" created in Step 1.
It finds the avatar that looks most like the person's available data (e.g., their text and video).
It then "borrows" the missing audio shoe from that matching avatar and gives it to the person.
It does this step-by-step until the person has a complete set of shoes (a complete set of data).

3. Why This is Special

It's Flexible: You can have 2 types of data (text/image) or 5 types (like sleep signals), and this method works for any number. It doesn't need a new design for every new puzzle.
It's Light: The "filling in" part uses frozen, pre-trained tools (like a library of existing knowledge) and doesn't need heavy computing power. It separates the "filling in" job from the "solving" job.
It Handles Chaos: The paper tested this in extreme scenarios where more than 50% of the data was missing. Even when the AI was missing half its senses, it still performed incredibly well.

4. The Results: Beating the Competition

The authors tested UL4M4 on three very different "puzzles":

Movie Genres (MM-IMDb): Guessing if a movie is a Comedy or Drama using posters and text.
Movie Moods (CMU-MOSI): Guessing if a movie review is happy or sad using text, audio, and video.
Sleep Stages (Sleep-EDF): Determining if a person is awake or in deep sleep using 5 different brain signal types.

The Big Win:
In the hardest tests (where data was severely missing), UL4M4 achieved scores above 0.7 (a very high mark) consistently. This is the first time a method has done this on the difficult "Movie Moods" dataset under such bad conditions. It beat all the previous "state-of-the-art" methods, even those specifically designed just for that one task.

5. The "Secret Sauce" (Cluster Size)

One might worry: "What if we group people into 10 groups or 100 groups? Does it matter?"
The paper found that it doesn't matter much. Whether the AI creates 20 groups or 100 groups, the results stay stable. This makes the tool very easy to use because you don't need to be a math genius to tune it perfectly.

Summary

Think of UL4M4 as a universal translator for missing data. Instead of letting a broken puzzle stop the game, it looks at the patterns of the pieces you do have, finds a similar group, and confidently fills in the blanks. This allows the AI to solve the final problem (like predicting a movie's mood) accurately, even when the input data is incomplete, messy, or missing half its parts.

Technical Summary: Unsupervised Learning for Missing Modalities in Multimodal Learning (UL4M4)

1. Problem Statement

Multimodal learning leverages complementary information from diverse data sources (e.g., text, audio, vision) to enhance robustness and accuracy in tasks like sentiment analysis and classification. However, real-world applications frequently suffer from missing modalities due to sensor failures, privacy restrictions, or acquisition constraints. Traditional multimodal models, which assume all modalities are present during both training and inference, suffer severe performance degradation when faced with incomplete data.

Existing solutions generally fall into two categories:

Generative methods: Attempt to reconstruct missing modalities from available ones (e.g., via adversarial learning or diffusion).
Non-generative methods: Learn shared representations or adapt fusion strategies (e.g., mixture-of-experts, retrieval-augmented learning).

Despite progress, current approaches face significant limitations: they often support only a fixed number of modalities, handle specific missing patterns (e.g., only one missing modality), tightly couple imputation mechanisms with downstream tasks (limiting generalizability), or lack rigorous evaluation under severe missing-data regimes (e.g., >50% missing slots).

2. Methodology: UL4M4 Framework

The authors propose UL4M4 (Unsupervised Learning for Missing Modalities in Multi-Modal Learning), a flexible, task-independent framework that imputes missing feature embeddings before supervised prediction. The framework operates in two distinct stages using frozen pre-trained encoders, ensuring the imputation module is lightweight and decoupled from the downstream task.

Stage 1: Multi-Modal Clustering with Partial-Modality Distance

The first stage performs unsupervised clustering on the training set to capture cross-modal structures from incomplete observations.

Modality-Specific Normalization: For each modality $m$ , the system calculates mean ( $\mu_m$ ) and standard deviation ( $\sigma_m$ ) over available embeddings. Features are normalized via z-score: $\tilde{z}^{(m)}_i = (z^{(m)}_i - \mu_m) / \sigma_m$ .
Partial-Modality Distance Metric: A novel distance metric is introduced to handle samples with different sets of available modalities. For a sample $i$ and cluster center $j$ , the distance $d(i, j)$ is computed only over the set of shared modalities $M_{i,j}$ :
$d(i, j) = \sqrt{\frac{1}{|M_{i,j}|} \sum_{m \in M_{i,j}} \frac{\|\tilde{z}^{(m)}_i - \tilde{c}^{(m)}_j\|^2_2}{d_m}}$
where $d_m$ is the dimensionality of modality $m$ . This metric ensures scale invariance across varying dimensionalities and prevents bias toward centers with more overlapping modalities.
Clustering Process: K-means clustering is performed iteratively. Cluster centers inherit the modalities of their initializing samples. If a center lacks a specific modality, that modality remains undefined for that center.

Stage 2: Iterative Greedy Completion

Once cluster centers are established, missing modalities for any incomplete sample are imputed via an iterative greedy process:

Candidate Generation: For a sample $i$ , a set of candidate completions is created by combining the sample's available modalities with the corresponding modalities from each of the $k$ cluster centers.
Iterative Selection: In each iteration, the algorithm computes the partial-modality distance between every remaining candidate and all cluster centers.
Imputation: The candidate-center pair with the minimum distance is selected. The missing modalities present in the selected candidate are copied into the target sample's representation.
Termination: This process repeats until the sample possesses embeddings for all $K$ modalities. The method guarantees termination because the union of all cluster centers collectively covers all modalities present in the training data.

Downstream Task Training

After imputation, all training samples possess complete modality sets. A fusion module (using Multi-Head Self-Attention) combines these representations, followed by a task-specific head (e.g., classification or regression). Only the fusion and head parameters are trained; encoders and imputed embeddings remain frozen. At inference, test samples with missing modalities undergo the same two-stage imputation using statistics derived solely from the training set.

3. Key Contributions

The paper highlights five primary contributions:

Arbitrary Modality Support: The framework supports datasets with any number of input modalities without requiring architectural redesign.
Arbitrary Missing Patterns: It handles any combination of missing modalities at the sample level, including extremely severe cases where >50% of slots are missing.
Task-Independent Imputation: The unsupervised imputation stage is decoupled from the supervised task, allowing the component to be reused or extended to other multimodal problems.
Rigorous Evaluation: Experiments are conducted under highly incomplete regimes, frequently exceeding 50% missing modality slots.
State-of-the-Art Performance: The method achieves consistent F1-Micro scores above 0.7 on challenging missing configurations in multimodal sentiment analysis, outperforming existing baselines.

4. Experimental Results

The authors evaluated UL4M4 on three diverse benchmarks: MM-IMDb (movie genre classification), CMU-MOSI (multimodal sentiment analysis), and Sleep-EDF (polysomnographic sleep staging).

Performance: UL4M4 consistently outperformed strong baselines (including M3Care, ShaSpec, ICL-CA, SimMLM, and the task-specific MissModal) across all datasets and missing configurations.
- On MM-IMDb, it achieved significantly higher F-Micro scores (e.g., 0.92 vs. ~0.58 for the next best) across all missing patterns.
- On CMU-MOSI, it achieved F1-Micro scores consistently above 0.7 even when more than 50% of modality slots were missing, a threshold where other methods struggled.
- On Sleep-EDF (5 modalities), it demonstrated graceful degradation as missing rates increased, outperforming baselines in high-missing regimes.
Statistical Significance: Differences were confirmed as statistically significant via t-tests and Wilcoxon signed-rank tests.
Robustness to Hyperparameters: The method showed remarkable stability regarding the number of clusters ( $k$ ). Performance remained consistent across $k$ values ranging from 10 to 100, suggesting that careful tuning of $k$ is not required.
Qualitative Analysis: t-SNE visualizations of imputed embeddings showed that the generated features tightly overlapped with ground-truth distributions, preserving realistic semantic structures.

5. Significance and Claims

The paper claims that UL4M4 represents a significant advancement in handling real-world incomplete multimodal data. Its primary significance lies in its generality and robustness:

It provides a unified representation strategy that does not rely on task-specific inductive biases or complex generative models.
It is the first method reported to achieve consistent F1-Micro scores above 0.7 on the challenging CMU-MOSI benchmark under severe missing-modality conditions.
By decoupling imputation from the downstream task, it offers a lightweight, plug-and-play solution that can be integrated with any fusion architecture, making it highly practical for heterogeneous and unreliable real-world environments where data completeness cannot be guaranteed.

The authors conclude that UL4M4 sets a new reference for handling missing modalities, offering a scalable solution that maintains high performance even when the majority of input data is missing.