AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuning

Technical Summary: AOEPT

Problem Statement
Deploying multimodal systems in real-world environments often necessitates handling scenarios where one or more modalities are missing (e.g., sensor failures, transmission errors). While recent approaches address this challenge for Multimodal Transformers (MTs) using prompt tuning, the authors identify a fundamental limitation termed the Implicit Modality-Reduction (IMR) Bottleneck. Existing prompt-tuning methods typically condition prompts solely on the observed modalities (or use random initialization). Consequently, these methods inadvertently restrict the MT's reasoning scope to the subspace of the remaining modalities. This effectively degrades a multimodal problem into a unimodal one, cutting off the model's access to the latent information sources of the missing modalities that were learned during pretraining.

Methodology: AOEPT
To overcome the IMR bottleneck, the authors propose AOEPT (Adaptive missing-adaptive modAl-cOntExtualized promPTing), a framework that shifts the paradigm from passively adapting to degraded inputs to actively compensating for missing information. The method operates through three primary stages:

1. Modal-Contextualized Prompt (MCP) Construction:
   Instead of random initialization, AOEPT distills global, modality-wise priors from the training data. It forwards modality-available training samples through frozen MT layers to collect layer-wise token representations. These representations are grouped via K-means clustering to form semantic prototypes, creating a lightweight, modality-specific information repository (e.g., Text-Contextualized Prompts or TCPs). These MCPs serve as latent repositories containing the global contextual distribution of each modality.

2. Instance-Aware Prompt Instantiation:
   When processing an incomplete sample (e.g., an image-only sample), AOEPT adaptively fetches the corresponding MCPs. These global prompts are then instantiated into instance-aware prompts conditioned on the remaining observed modalities. This is achieved via a gating mechanism (using an MLP and sigmoid function) that selectively activates the specific information within the MCPs most relevant to the current sample. To ensure the instantiated prompts remain consistent with the modality space, an Intra-Modal Latent Consistency Regularization (LCR) is applied, aligning the prompt representations with the latent representations of the available modalities.

3. Missing-Adaptive Prompt Tuning:
   The instantiated prompts are inserted into the MT encoder layers. For the initial layers, prompts are newly instantiated for each layer; for subsequent layers, they are propagated. Only the MCPs and the classification head are tuned, keeping the backbone frozen. This process explicitly supplements missing-modality information for each sample, restoring the MT's reasoning scope beyond the observed-modality-only subspace.

Key Contributions

• Identification of the IMR Bottleneck: The paper rethinks existing prompt-tuning paradigms, demonstrating that they unintentionally confine MT reasoning to modality-reduced subspaces, thereby failing to leverage pre-trained multimodal capacities.
• AOEPT Framework: A conceptually novel solution that explicitly restores access to missing-modality information repositories via a lightweight, modal-contextualized prompting fashion. It expands the reasoning scope of MTs beyond the remaining modalities without requiring heavy reconstruction networks or external retrieval modules.
• Diagnostic Metric (NM2I): The authors introduce Normalized Missing-modality Mutual Information (NM2I) to quantify the severity of the IMR bottleneck. This metric measures the mutual information between prompt tokens and the latent representations of the missing modality, providing empirical evidence of how well a method accesses missing information.
• Modality Information Scaling: The study reveals a "Modality Information Scaling Bottleneck" in existing methods, where performance plateaus even when training conditions improve (i.e., more complete data is available). In contrast, AOEPT benefits from additional training information to form more comprehensive global repositories.

Experimental Results
Experiments were conducted on three dual-modal benchmarks (MM-IMDb, HateMemes, Food101) and a tri-modal benchmark (IEMOCAP) using various backbones (CLIP, ViLT, MulT).

• Performance: AOEPT consistently outperforms strong baselines (including MAPs, DCP, MemPrompt, SyP, and RAGPT) across various missing rates (70% and 90%) and missing scenarios (text-only, image-only, or both missing).
• Efficiency: AOEPT achieves these gains with minimal computational overhead and a small number of learnable parameters, avoiding the high costs associated with memory mechanisms or instance-wise retrieval found in other methods.
• Ablation: Removing MCPs, instantiation, or consistency regularization leads to significant performance drops, confirming the necessity of each component.
• NM2I Analysis: Baseline methods exhibit near-zero NM2I, confirming they do not access missing-modality information. AOEPT achieves significantly higher NM2I, validating its ability to bridge the IMR bottleneck.

Significance and Claims
The paper claims that AOEPT reframes modality-missing learning for MTs from a passive adaptation to degraded input structures into an active information-access perspective. By internalizing a self-retrieval mechanism that distills global contextual knowledge into lightweight prompts, AOEPT allows MTs to implicitly access and leverage information from missing modalities without altering the backbone architecture or introducing heavy reconstruction modules. The authors position this as a principled and efficient solution that maintains the standard lightweight prompt-tuning paradigm while overcoming the inherent limitations of current approaches. The work is supported by the National Natural Science Foundation of China.