🤖 AI

Deep Multimodal Learning with Missing Modality: A Survey

This survey provides the first comprehensive review of deep learning methods for Multimodal Learning with Missing Modality (MLMM), detailing its motivations, distinctions from standard setups, current techniques, applications, datasets, and future challenges.

Original authors: Renjie Wu, Hu Wang, Hsiang-Ting Chen, Gustavo Carneiro

Published 2026-02-05

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Renjie Wu, Hu Wang, Hsiang-Ting Chen, Gustavo Carneiro

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Five Senses" Problem

Imagine you are trying to understand a movie. Usually, you have two main inputs: sight (the video) and sound (the dialogue and music). This is like a "multimodal" system—it uses multiple senses to get the full story.

However, in the real world, things go wrong. Maybe your speakers break (no sound), or the screen goes black (no video). Maybe you are in a foggy forest where you can't see, but you can hear a rustle. This is the Missing Modality problem.

Most AI models are like students who only know how to study when they have their textbook and their audio notes. If you take away one, they panic and fail. This survey paper is a massive guidebook for researchers on how to teach AI to stay calm and perform well even when it loses one of its "senses."

What is This Paper About?

This paper is a survey. Think of it as a librarian who has read every single book (354 papers!) written between 2012 and 2025 on this specific topic. The authors, Renjie Wu and his team, organized all these different solutions into a clear map so researchers know what works, what doesn't, and where to go next.

They call this field MLMM (Multimodal Learning with Missing Modality).

The Two Main Strategies: "Fixing the Data" vs. "Fixing the Brain"

The authors divide all the solutions into two big camps, like fixing a broken car either by repairing the parts or changing how the driver drives.

1. Data Processing: "Fixing the Parts"

This approach tries to fill in the missing information before the AI even starts thinking.

The "Blank Page" Method (Modality Composition): Imagine you are reading a book, but a page is torn out. You could just leave a blank white space (zeros) or scribble random nonsense (random values) on that page. The AI learns to ignore the blank space and focus on the rest. It's simple, but not very smart.
The "Copy-Paste" Method (Retrieval): If a page is missing, you look at other copies of the same book in the library, find the matching page, and paste it in. This works well if the books are identical, but if the stories are slightly different, you might paste the wrong scene in.
The "Imagination" Method (Modality Generation): This is the most advanced. The AI acts like a creative writer. If the audio is missing, the AI looks at the video and imagines what the sound should be, then creates it. It's like a magician pulling a rabbit out of a hat. The paper notes that while this is cool, it can sometimes "hallucinate" (make things up that aren't true).

2. Strategy Design: "Fixing the Brain"

Instead of trying to fix the missing data, this approach changes the AI's architecture so it can handle missing data naturally.

The "Spotlight" Method (Attention): Imagine a teacher in a classroom. If a student is absent, the teacher doesn't stop the lesson; they just shine their spotlight on the students who are there. "Attention" mechanisms allow the AI to dynamically focus only on the available data and ignore the missing parts.
The "Tutor" Method (Distillation): Imagine a smart student (the Teacher) who has all the books. A less smart student (the Student) only has half the books. The Teacher explains the missing chapters to the Student. The Student learns to understand the whole story even without the physical book.
The "Teamwork" Method (Model Combinations): Instead of one giant AI, you have a team of specialists. If the video is missing, you ask the "Audio Expert." If the audio is missing, you ask the "Video Expert." They vote on the answer together.
The "Super-Brain" (Large Language Models): Recently, huge AI models (like the ones powering chatbots) have become so smart they can understand text, images, and sound all at once. They are so flexible that if you take away one input, they just adapt and keep going, almost like a human who can still tell a story even if they can't see the picture.

Where is This Used? (The Real-World Examples)

The paper lists many places where this is critical:

Medical Scans: A doctor might have an MRI scan but no CT scan for a patient. The AI needs to diagnose the illness using just the MRI without guessing wildly.
Self-Driving Cars: A car's camera might get blinded by snow, or its radar might break. The car's AI needs to keep driving safely using only the sensors that are still working.
Emotion Detection: A video call might have bad audio but clear video. The AI should still be able to tell if you are happy or sad by looking at your face.
Robotics: A robot exploring a cave might lose its GPS signal. It needs to navigate using only its cameras and touch sensors.

The Problems We Still Haven't Solved

Even though we have these cool tricks, the paper points out some major headaches:

The "Fake Data" Trap: When AI "imagines" missing data, it sometimes makes up details that are wrong. If a self-driving car imagines a road where there is a cliff, that's dangerous.
The "Lazy" AI: Sometimes, even if the AI tries to fill in the missing piece, it ends up ignoring the new info and just relying on the one sensor it does have, which might not be enough.
The "Messy Library": There is no single standard test. One researcher might test their AI with 10% missing data, while another tests with 90%. It's hard to compare who is actually the best.
Too Heavy: Many of these solutions require massive computer power (like a supercomputer). But real-world devices (like a smartwatch or a drone) are small and have weak batteries. We need "lightweight" solutions that work on small chips.

The Bottom Line

This paper is a roadmap. It tells us that while we have made great progress in teaching AI to cope with broken sensors and missing data, we still need better ways to do it without making things up, without needing super-computers, and without getting confused by messy real-world situations. The goal is to build AI that is as robust as a human: able to understand the world even when the view is blurry or the microphone is broken.

Technical Summary: Deep Multimodal Learning with Missing Modality: A Survey

1. Problem Definition

The paper addresses the Missing Modality Problem in Deep Multimodal Learning. In real-world scenarios, multimodal systems frequently encounter incomplete data due to sensor malfunctions, hardware limitations, privacy constraints, data loss, or environmental interference. Unlike conventional multimodal learning, which assumes all $N$ modalities are consistently available during training and testing, Multimodal Learning with Missing Modality (MLMM) aims to develop models that remain robust and performant when one or more input modalities are absent. The core challenge is to effectively exploit available modalities while maintaining performance comparable to systems trained with complete information, without degrading due to the absence of critical inputs.

2. Methodology Taxonomy

The authors propose a novel, fine-grained taxonomy categorizing existing deep MLMM methods into two primary aspects: Data Processing and Strategy Design. These aspects are further divided into four types and twelve sub-categories (summarized in Table 1 of the paper).

A. Data Processing Aspect

This aspect focuses on how the model handles missing information at the data or representation level.

Modality Imputation: Operates at the raw data level to fill in missing information.
- Modality Composition: Replaces missing data with zero/random values or retrieves/copies data from similar instances (e.g., K-Nearest Neighbors). While simple, these methods often fail at pixel-level tasks and may introduce overfitting or duplicate samples.
- Modality Generation: Uses generative models (Auto-Encoders, GANs, Diffusion Models) to synthesize missing modalities from available ones. Individual generators are trained per modality, while unified generators attempt to produce all missing modalities simultaneously. These methods improve data diversity but increase computational and storage costs.
Representation-Focused Models: Operates at the feature representation level.
- Coordinated-Representation: Imposes constraints (e.g., Tensor Rank Regularization, Deep Canonical Correlation Analysis) to align available modalities in a shared semantic space, ensuring semantic consistency even with missing inputs.
- Representation Composition: Fuses available representations using arithmetic operations (pooling, addition) or retrieval-based methods to form a complete representation without learnable parameters.
- Representation Generation: Uses generative models to create missing feature representations directly.
  - Indirect-to-Task: Treats modality reconstruction as an auxiliary task during training (often using Encoder-Decoder architectures) but discards the generator during inference.
  - Direct-to-Task: Directly maps available representations to missing ones using small generative networks (e.g., Hallucination Networks, Bayesian learning) without an explicit reconstruction step.

B. Strategy Design Aspect

This aspect focuses on architectural adjustments and model combinations to adapt to varying input conditions.

Architecture-Focused Models: Designs flexible architectures that adapt to the number of available modalities.
- Attention-Based: Utilizes self-attention mechanisms (intra- and inter-modality) and masking strategies to dynamically fuse available modalities. Includes Transformer-based Joint Representation Learning (JRL) and Parameter-Efficient Learning (PEL) methods like Prompt and Adapter tuning.
- Distillation-Based: Transfers knowledge from a "teacher" model (trained on full modalities) to a "student" model (handling missing modalities). Includes response distillation (logits), intermediate feature distillation, and process-based methods like Mean Teacher and Self-Distillation.
- Graph Learning-Based: Models relationships between modalities and samples using Graph Neural Networks (GNNs) to dynamically fuse available features and compensate for missing ones via graph completion or message passing.
- Multimodal Large Language Models (MLLMs): Leverages foundation models (e.g., LLaVA, AnyGPT) where the LLM acts as a feature processor capable of accepting variable numbers of modality tokens, naturally handling missing inputs through their flexible architecture.
Model Combinations: Employs strategies leveraging multiple specialized models.
- Ensemble Methods: Combines predictions from multiple models (multimodal or unimodal) via voting or weighted averaging.
- Dedicated Training: Trains specific models for specific missing-modality combinations (e.g., a model for Modality 1+3 only) and selects the appropriate one during inference.
- Discrete Scheduler: Uses an LLM as a controller to orchestrate and select downstream modules based on available modalities and task instructions.

3. Key Contributions

The paper makes three primary contributions:

Comprehensive Survey: It provides the first extensive review of deep MLMM methods across diverse domains (Affective Computing, Medical, Remote Sensing, Robotics, etc.), accompanied by a compilation of relevant datasets.
Novel Taxonomy: It introduces a fine-grained classification framework based on Data Processing (Imputation vs. Representation) and Strategy Design (Architecture vs. Combinations), offering a structured way to analyze existing literature.
Critical Analysis: It offers an in-depth analysis of current approaches, highlighting their trade-offs (e.g., generative methods vs. efficiency, recovery vs. non-recovery), and identifies open challenges and future research directions.

4. Results and Statistics

The survey analyzes 354 significant papers published between 2012 and August 2025. Key statistical findings include:

Method Distribution: Approximately 75.5% of works focus on recovering missing modality information (early, intermediate, or late-stage recovery), while 24.5% focus on inference without recovery (non-recovery).
Recovery Stages: Intermediate-stage recovery (feature-level) is the most prevalent (45.8%), followed by early-stage (20.3%) and late-stage (4.7%).
Training Data Constraints: A significant gap exists in handling incomplete training datasets. Only 36.4% of reviewed works trained on datasets where missing modality data was unavailable during training; the majority (63.6%) assumed access to full data during training and only simulated missingness during testing.
Technical Evolution: The field has evolved from early heuristic data-level fixes and ensemble methods to representation learning, flexible architecture design (attention, distillation), and recently, foundation-model-based approaches.

5. Significance and Claims

The authors claim this survey is the first comprehensive review specifically focused on deep multimodal learning with missing modalities. Its significance lies in:

Unifying the Field: Providing a structured taxonomy that bridges disparate methods from computer vision, NLP, and signal processing.
Highlighting Real-World Relevance: Emphasizing that missing modalities are not just synthetic challenges but inherent to real-world deployments (e.g., sensor failures in space exploration, privacy in medical AI).
Identifying Critical Gaps: The paper modestly but firmly points out that current benchmarks often fail to model realistic missingness patterns (e.g., asynchronous sensor failures) and that most methods struggle with incomplete training datasets.
Future Directions: It calls for the development of unified benchmarks (similar to MultiBench), more efficient and lightweight models for edge devices, and robust methods for streaming data and scientific applications where data is inherently incomplete.

The paper concludes that while progress has been made, the field requires more rigorous evaluation protocols, standardized benchmarks, and architectures explicitly designed for the structural and persistent missingness found in real-world environments.