💻 computer science

Conservative Proxy Prompting for Reliable Frozen Vision–Language Models with Missing Modalities

This paper proposes Conservative Proxy Prompting (CPP), a parameter-efficient framework that enhances the reliability of frozen vision-language models under missing modalities by constructing feature-space proxy representations and conservatively regulating their contribution to avoid misleading signals.

Original authors: Xiaoyong Mei, Wei Ji, Jiale Dong, Chao Duan, Mingyan Zhang, Fudan Zheng

Published 2026-09-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaoyong Mei, Wei Ji, Jiale Dong, Chao Duan, Mingyan Zhang, Fudan Zheng

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, there is a growing class of systems known as vision-language models. These are digital minds trained to understand the world by looking at pictures and reading text simultaneously, learning how images and words relate to one another. They have become remarkably skilled at tasks like describing a photo or answering questions about a scene, largely because they are trained on massive collections of paired images and text. However, these systems are built on a fragile assumption: that they will always receive both the picture and the description at the same time. In the messy reality of the real world, this is rarely guaranteed. Sensors fail, data gets lost in transmission, and privacy filters might strip away a description, leaving the system with only half the information it expects. When a model trained on complete pairs is suddenly forced to guess with only one piece of the puzzle, its confidence often crumbles, and its answers become unreliable. The challenge for researchers is not just to help the model guess what is missing, but to teach it to know the difference between what it actually sees and what it is merely inferring.

A team of researchers at Zhejiang Normal University and Sun Yat-Sen University has developed a new method to solve this problem, called Conservative Proxy Prompting. Their approach acknowledges a fundamental truth about artificial intelligence: when a model has to guess a missing piece of information based on what it can see, that guess is inherently uncertain. Imagine a person trying to identify a dish in a restaurant. If they can see the food and read the menu, they are certain. If the menu is missing, they can look at the food and guess the name, but they should remain slightly unsure of their answer. Existing methods often try to reconstruct the missing information as if it were real, effectively treating a guess with the same weight as a direct observation. The researchers argue that this is dangerous because the reconstructed data is just an estimate, and treating it as fact can mislead the system. Instead, their new framework treats these guesses as "proxy" evidence—useful, but requiring a cautious hand.

The core of their solution involves a two-step strategy that works with the existing, powerful AI models without needing to retrain them from scratch. First, the system uses a technique called prompt learning to gently nudge the model's understanding toward the specific task at hand, like identifying movie genres or food types, using only a tiny number of adjustable settings. This allows the model to stay sharp and ready for the specific job without overhauling its entire brain. Second, and most importantly, the system introduces a mechanism to handle missing data. When a picture is missing, the system uses the text to create a rough sketch of what the picture might look like. When the text is missing, it uses the picture to guess the words. But here is the crucial difference: before this guessed information is mixed with the real data, the system deliberately shrinks its influence. It applies a "conservative" filter, reducing the weight of the guess so that it supports the decision without overpowering the direct evidence. This ensures that if the guess is wrong, it doesn't drag the final answer down with it.

To test this idea, the researchers put their method through its paces on three very different real-world datasets. They used a collection of movie posters and plot summaries, a massive set of food images with recipes, and a challenging set of internet memes designed to test if a system can spot hate speech. In each case, they simulated real-world failures by randomly removing either the image or the text from the data, sometimes removing information from up to ninety percent of the samples. The results were clear: the new method consistently outperformed other approaches that tried to simply reconstruct the missing data. By keeping the influence of the guesses in check, the system maintained high accuracy even when it was missing most of its information. It proved that a model does not need to be perfect at guessing to be reliable; it just needs to know how much to trust its own guesses.

The study also revealed that this reliability comes without a heavy cost. The new method required only a very small number of adjustable parameters—roughly between 1.6 and 2.7 million—to achieve these results. In comparison, other methods that tried to achieve similar robustness often required nearly three times as many adjustable settings. This efficiency is significant because it means the system can be adapted to new tasks quickly and stored easily, without needing massive computing power. The researchers found that the specific number of "nudge" settings mattered, but there was no single magic number that worked for every situation; a moderate amount of adjustment was generally the sweet spot. Furthermore, the system showed it could handle situations it had never seen before. Even when trained only on complete data and then tested on missing data, it adapted well, suggesting that the principle of being cautious with guesses is a robust strategy that generalizes beyond the specific training examples.

Ultimately, this work shifts the focus from trying to perfectly recreate missing information to managing the uncertainty of that recreation. The researchers demonstrated that in a world where data is often incomplete, the most reliable AI systems are not necessarily the ones that try to fill every gap, but the ones that understand the limits of their own knowledge. By treating inferred information as a helpful but secondary voice, rather than an equal partner to direct observation, the system becomes more stable and trustworthy. This approach offers a practical path forward for deploying intelligent systems in the real world, where sensors break and data is lost, ensuring that these tools remain useful even when they are not seeing the whole picture.

Technical Summary: Conservative Proxy Prompting for Reliable Frozen Vision–Language Models with Missing Modalities

1. Problem Statement

Frozen vision–language models (VLMs), such as CLIP and SigLIP, have demonstrated strong transferability by learning aligned image–text representations from large-scale pretraining. However, their reliability degrades significantly in practical scenarios where one modality (image or text) becomes unavailable due to privacy restrictions, sensor failures, or transmission interruptions.

Existing approaches to missing modalities generally fall into two categories:

Modality Completion: These methods attempt to reconstruct missing inputs or latent representations (e.g., via diffusion or masked learning). While they provide complementary information, the reconstructed features are estimates that may contain uncertain signals or semantic inconsistencies, potentially introducing misleading evidence into the reasoning process.
Missing-Aware Adaptation: These methods incorporate modality availability into architectures or prompt learning strategies. However, most existing prompt-based approaches assume complete observations or fail to explicitly distinguish between the reliability of directly observed evidence and inferred proxy representations.

The core challenge addressed by this work is how to adapt frozen VLMs to incomplete observations without treating uncertain, inferred proxy features as equivalent to genuine, observed evidence, thereby preventing the propagation of unreliable information during multimodal fusion.

2. Methodology: Conservative Proxy Prompting (CPP)

The authors propose Conservative Proxy Prompting (CPP), a parameter-efficient framework designed to adapt frozen VLMs under incomplete observations. CPP separates task adaptation from proxy reliability through three sequential stages:

2.1 Lightweight Task-Aware Prefix (LTP)

LTP introduces task-related semantic guidance into the textual input before prompt adaptation. It applies a lightweight transformation to the input description and processes it through the original CLIP tokenizer. Crucially, LTP modifies the textual input representation without introducing additional trainable parameters or updating the frozen text encoder, preserving pretrained knowledge while providing dataset-level context.

2.2 Dual-Stream Soft Prompt Adaptation (D-SPA)

D-SPA adapts both the visual and textual branches of the frozen CLIP backbone using modality-specific learnable soft prompts:

Textual Branch: Learnable prompt tokens are added via residual addition to the token embedding sequence.
Visual Branch: Learnable visual prompt tokens are inserted into the image encoder input sequence alongside the class token and patch embeddings.
This stage optimizes a small number of parameters to align the frozen representation space with downstream tasks while maintaining the original CLIP encoding mechanism.

2.3 Lightweight Missing-Aware Reliability Control (LMRC)

This is the core innovation for handling missing modalities. When a modality $r$ is unavailable, LMRC constructs a feature-space proxy $\hat{z}_i^r$ from the observed modality using a directional affine projector (e.g., $W_{I \to T}z_i^I + b_{I \to T}$ ).

Unlike completion-based methods that optimize for explicit recovery, LMRC treats the generated proxy as uncertain auxiliary evidence. To regulate its contribution, a conservative attenuation operation is applied before multimodal fusion:
$z_i^r = (1 - \delta_i^r)z_i^r + \alpha \delta_i^r \hat{z}_i^r$
Where:

$\delta_i^r$ is an indicator for whether modality $r$ is missing.
$\alpha$ is a fixed conservative coefficient (set to 0.1 in experiments) that scales down the contribution of the inferred proxy.
The observed modality features are preserved unchanged.

Additionally, a missing-type embedding ( $E_m[m_i]$ ) is concatenated with the fused features to explicitly inform the prediction head of the modality availability state, allowing the model to distinguish between direct observations and proxy-assisted representations.

2.4 Optimization

The framework uses a lightweight two-layer MLP as the prediction head. Optimization relies solely on the standard cross-entropy (single-label) or balanced binary cross-entropy (multi-label) objectives. No auxiliary reconstruction losses, adversarial objectives, or proxy supervision are used. Only the lightweight parameters (prompts, projectors, embeddings, and head) are updated; the CLIP backbone remains frozen.

3. Key Contributions

The paper outlines three primary contributions:

Framework Proposal: The introduction of CPP, a parameter-efficient framework enabling reliable adaptation of frozen VLMs under diverse incomplete observation conditions.
Reliability-Aware Strategy: A novel strategy that constructs lightweight feature-space proxies and conservatively controls their contribution via attenuation before fusion. This explicitly reduces the influence of uncertain inferred information, distinguishing it from observed evidence.
Empirical Validation: Extensive experiments demonstrating robust performance across text-missing, image-missing, and mixed-missing settings with minimal trainable parameters.

4. Experimental Results

The authors evaluated CPP on three benchmarks: MM-IMDb (multi-label movie genre), Food-101 (fine-grained food recognition), and Hateful Memes (binary classification).

Performance: CPP achieved the best results in 25 out of 27 dataset–missing-rate–modality combinations compared to strong baselines (MMP, MaPLe, DePT, DCP, SyP). It showed consistent improvements, particularly under high missing rates (70% and 90%).
Parameter Efficiency: CPP requires only 1.6–2.7M trainable parameters, significantly fewer than the ~5.8M parameters in the closest baseline (SyP), yet achieved higher average performance in most settings.
Ablation Studies: Removing the conservative attenuation (CA) led to performance degradation, confirming that uncontrolled proxy features introduce noise. Removing prompt adaptation (PL) or the task-aware prefix (LTP) also resulted in drops, validating the necessity of both representation adaptation and semantic guidance.
Generalization: When trained on complete pairs and tested on missing-modality scenarios (without missing-state augmentation during training), CPP maintained stable performance, demonstrating strong generalization to unseen missing patterns.
Visualization: UMAP visualizations showed that CPP learns structured multimodal feature distributions even under missing conditions. Cross-modal retrieval analysis indicated that the model preserves meaningful semantic alignment while reflecting the uncertainty of transferred information.

5. Significance and Claims

The paper claims that the significance of CPP lies in its ability to address the reliability gap in missing-modality learning. By explicitly separating the generation of proxy information from its utilization, CPP avoids the pitfall of treating reconstructed estimates as ground truth.

The authors emphasize that their approach does not rely on complex reconstruction mechanisms or large-scale parameter updates. Instead, it leverages a conservative attenuation mechanism to safely exploit complementary information from available modalities while mitigating the risk of unreliable feature propagation. The framework offers a practical solution for deploying frozen VLMs in real-world environments where data completeness cannot be guaranteed, achieving robustness with high parameter efficiency.

The paper concludes modestly, noting that future work could explore adaptive reliability estimation (dynamically adjusting $\alpha$ ) and extending the framework to larger foundation models, but asserts that the current fixed conservative coefficient provides a robust and effective approximation for diverse scenarios.