Learning from Reliable Latent Prompts for Visual Recognition with Missing Modalities

Technical Summary: Learning from Reliable Latent Prompts for Visual Recognition with Missing Modalities

Problem Statement
Large-scale Multimodal Models (LMMs) have achieved superior performance by synergizing information across diverse, massive-scale paired modalities. However, in real-world deployments, inputs are frequently incomplete due to privacy constraints, sensor failures, or data collection limitations. Existing models optimized for modality-complete data suffer precipitous performance degradation when faced with missing modalities. While recent prompt learning approaches attempt to mitigate this by generating dynamic prompts conditioned on instance-level features, these strategies are fundamentally flawed under high missing rates. As the proportion of incomplete modalities increases, the instance-level features used to condition the prompts become unreliable, biased, and unstable. This leads to dynamic prompts that collapse into unimodal observations, disrupting cross-modal alignment and failing to maintain consistent class-level structures.

Methodology: Learning from Reliable Latent Prompts (LLP)
To address the unreliability of input-conditioned strategies, the authors propose Learning from Reliable Latent Prompts (LLP), a paradigm that decouples prompt learning from corrupted inputs. Instead of relying on instance-level features, LLP models input-agnostic learnable prompts as stable latent anchors that encapsulate modality-intrinsic priors.

The framework, built upon a frozen CLIP backbone, operates through three core mechanisms:

1. Modality-Specific Latent Anchors: LLP introduces learnable prompt tokens ($P_{I,S}$ for images and $P_{T,S}$ for text) that are input-agnostic. These anchors are derived from static prompts via a transformation function (FC + GELU + LN) and serve as stable foundations encoding modality-level priors, independent of the specific input sample or its missingness.
2. Dual Anchor-Induced Prompts (DAIP): To enable robust cross-modal interaction without relying on incomplete inputs, the framework employs a dual-path cross-attention mechanism. The text anchor queries the image anchor (and vice versa) to exchange complementary information. This interaction generates refined, dual anchor-induced prompts ($P_{I,D}$ and $P_{T,D}$) through a lightweight MLP and normalization. This process allows the model to approximate missing semantic information from a stable latent space.
3. Layer-wise Prompt Generation: The final prompts for each modality are constructed by concatenating the latent anchors and the induced prompts. These are injected into every layer of the encoder, allowing for progressive refinement and hierarchical representation learning across the network depth.

During training, only the prompt parameters and a task-specific fully connected layer are optimized, while the backbone remains frozen.

Key Contributions

• A Novel Paradigm: The paper proposes LLP, a latent prior-driven prompting framework that decouples prompt learning from unreliable inputs, capturing stable modality-level priors to improve robustness under incomplete observations.
• Input-Agnostic Design: The introduction of modality-specific latent anchors and a dual-path cross-attention mechanism enables input-independent prompt learning and stable cross-modal interaction, facilitating semantic compensation even under extreme missing rates (e.g., 90%).
• State-of-the-Art Performance: The approach demonstrates superior performance across multiple benchmarks and missing-modality scenarios, validating the efficacy of learning from reliable latent prompts over input-conditioned methods.

Experimental Results
The authors evaluated LLP on three benchmark datasets: MM-IMDb (movie genre prediction), UPMC Food-101 (food classification), and Hateful Memes (hate speech detection). Experiments were conducted under varying missing rates ($\eta = 50\%, 70\%, 90\%$) and missing scenarios (missing-text, missing-image, missing-both).

• Performance: LLP consistently outperformed six state-of-the-art baselines (including CoOp, MMP, MaPLe, DePT, DCP, and SyP) across all datasets and missing settings.
• High Missing Rates: The performance gains were most pronounced under high missing rates (e.g., $\eta = 90\%$). For instance, on the Hateful Memes dataset, LLP achieved an AUROC of 70.90% at 90% missing text, significantly outperforming the next best method (SyP at 69.70%).
• Ablation Studies: Experiments confirmed that the modality-specific latent anchors alone provided substantial gains over baselines, and the addition of the dual anchor-induced prompts yielded further significant improvements.
• Generalization: Models trained under "missing-both" scenarios demonstrated strong generalization to specific missing scenarios (text-only or image-only), indicating flexible adaptation to diverse missing patterns.
• Visualization: Feature visualizations showed that LLP produces more compact intra-class distributions and clearer inter-class separation compared to input-conditioned methods, which often result in fragmented, modality-specific clusters.

Significance and Claims
The paper claims that LLP offers a simple yet effective solution to the missing-modality problem, a critical bottleneck in the practical utility of LMMs. By shifting the focus from unreliable, instance-level input features to stable, input-agnostic latent priors, the method provides a robust mechanism for cross-modal knowledge compensation. The authors emphasize that this paradigm is particularly valuable for real-world applications where data integrity cannot be guaranteed. Furthermore, the approach maintains high parameter efficiency by optimizing only a small set of learnable parameters, making it practical for deployment. The work establishes a promising direction for robust multimodal learning by demonstrating that reliable semantic reasoning can be achieved even when up to 90% of modalities are missing.