Technical Summary: VLMGuard

Problem Statement

Vision-Language Models (VLMs) are increasingly deployed in open-world environments where they process a mixture of benign and adversarially manipulated inputs. Unlike text-only models, VLMs face an expanded attack surface where adversaries can manipulate visual, textual, or both channels to induce unsafe behaviors, such as generating harmful content or executing unintended tool actions.

The primary challenge in developing safeguards for VLMs is the scarcity of high-quality, labeled datasets containing both benign and malicious prompts. Curating such data requires significant human effort and continuous updates as attack strategies evolve. Consequently, there is a critical need for methods that can effectively detect malicious prompts using the vast amounts of unlabeled user prompts naturally available in real-world deployments, without relying on extensive manual annotation.

Methodology: VLMGuard

The authors propose VLMGuard, a two-stage learning framework designed to bootstrap malicious prompt detectors from unlabeled data. The framework operates under the assumption that unlabeled user prompts follow a Huber contamination model, where the data distribution $P_{unlabeled}$ is a mixture of benign ($P_{benign}$) and malicious ($P_{malicious}$) distributions with a small mixing ratio $\pi$.

Stage 1: Automated Maliciousness Estimation via Latent Subspace

The first stage leverages the internal representations of a pre-trained VLM to distinguish between benign and malicious samples within the unlabeled mixture.

1. Representation Extraction: For each prompt $(x_v, x_t)$, a $d$-dimensional representation $f$ is extracted from a specific layer $\ell$ of the VLM.
2. Subspace Identification: The authors perform Singular Value Decomposition (SVD) on the centered embedding matrix. Theoretical analysis suggests that under the Huber contamination model, the leading singular directions of the representation cloud are biased toward the deviations caused by the rare malicious samples.
3. Scoring: A maliciousness estimation score $\kappa_i$ is computed for each prompt based on its projection energy onto the top-$k$ dominant singular directions, weighted by their singular values:
   $$\kappa_i = \frac{1}{k} \sum_{j=1}^{k} \lambda_j \cdot \langle f_i - \mu, v_j \rangle^2$$
   where $\mu$ is the empirical mean, $v_j$ are the right singular vectors, and $\lambda_j$ are the singular values.
4. Pseudo-Partitioning: Prompts are partitioned into a candidate malicious set $\mathcal{M}$ (high scores) and a candidate benign set $\mathcal{B}$ (low scores) using a threshold $T$. This creates a noisy pseudo-labeling of the unlabeled data.

Stage 2: Training the Safeguarding Prompt Classifier

The second stage addresses the brittleness of direct projection scores, which can be sensitive to surface-level variations (e.g., paraphrasing) that do not reflect malicious intent.

1. Classifier Training: A lightweight binary classifier (a 3-layer MLP) is trained on the pseudo-labeled sets $\mathcal{M}$ (label 1) and $\mathcal{B}$ (label 0) using logistic loss.
2. Robustness: The authors argue that this classifier learns to discount "nuisance variations" (such as lexical or syntactic changes in benign prompts) that spuriously correlate with the dominant subspace energy, thereby providing a more robust decision boundary for test-time detection.

Key Contributions

1. Framework Formulation: The introduction of VLMGuard, a two-stage framework that formalizes malicious prompt detection by leveraging unlabeled user prompts from the wild, offering a practical solution to the data scarcity problem.
2. Principled Scoring Function: The development of a scoring function derived from VLM representations (via SVD) that estimates the likelihood of a prompt being malicious. The paper provides theoretical justification showing that projection energy onto leading covariance directions creates a class-conditional gap proportional to the separation between benign and malicious means.
3. Empirical Validation: Extensive experiments demonstrating the framework's efficacy, scalability to larger models (e.g., 72B parameters), and robustness against realistic prompt variations and abnormal benign data.

Experimental Results

The authors evaluated VLMGuard on three real-world scenarios using five datasets (JailBreakV, GPT4V, VLGuard, MLLMGuard, MSSBench) and two backbone VLMs (LLaVA-1.6-7B and Qwen2.5-VL-7B/72B).

• Performance: VLMGuard achieved state-of-the-art performance across all scenarios. On average, it improved the Area Under the Receiver Operating Characteristic Curve (AUROC) by 5.39% over the best existing method (GPT-5.4) and significantly outperformed uncertainty-based and activation-based baselines.
• Robustness to Low Prevalence: The method remained effective even when the malicious ratio $\pi$ in the unlabeled training set was extremely low (e.g., $\pi=0.001$), achieving an AUROC of 88.16%.
• Generalization: The framework demonstrated strong transferability, maintaining high detection accuracy when trained on one dataset and tested on different malicious benchmarks.
• Ablation Studies:
  • Classifier Necessity: Directly using the SVD projection score yielded lower performance than the two-stage approach, confirming the classifier's role in denoising and handling prompt variations.
  • Layer Selection: Intermediate layers of the VLM provided the best representations for detection, outperforming both early and final layers.
  • Scalability: Performance improved when scaling from 7B to 72B parameter models.

Significance and Claims

The paper claims that VLMGuard offers a practical and flexible approach for real-world VLM deployment. By eliminating the need for extra human annotations and additional external models (such as separate judge models or diffusion models for denoising), the framework provides a self-contained solution for safeguarding VLMs.

The authors emphasize that their method is robust to realistic prompt variations (lexical and syntactic changes) and can distinguish malicious intent even in the presence of abnormal but benign data. The work aims to inspire future research on leveraging unlabeled prompt datasets for safety-critical applications, addressing the fundamental bottleneck of labeled data scarcity in adversarial machine learning.