💬 NLP

Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models

This paper introduces VEV-UAP, a cost-efficient and task-agnostic attack framework that exploits structurally concentrated vulnerabilities in the value components of middle-layer attention mechanisms within LVLM vision encoders to generate universal adversarial perturbations that achieve state-of-the-art success rates across models and tasks without requiring textual inputs.

Original authors: Hee-Seon Kim, Minbeom Kim, Seokil Ham, Changick Kim

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hee-Seon Kim, Minbeom Kim, Seokil Ham, Changick Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Vision-Language Model (LVLM) as a highly intelligent, two-person team solving puzzles together.

The Eyes (Vision Encoder): This partner looks at a picture and describes what they see.
The Brain (Language Model): This partner listens to the description and writes the final answer.

For a long time, researchers thought that to trick this team into giving the wrong answer, you had to mess with the entire "Eyes" partner. They assumed every part of the eyes was equally important and equally easy to fool. They tried to add a tiny bit of "static noise" to the whole image, hoping it would confuse the whole system.

The Paper's Big Discovery: The "Weak Link"
The authors of this paper realized that the "Eyes" partner isn't a uniform block of muscle; it's more like a complex factory with many different stations. They investigated the factory floor and found something surprising: not all stations are equally vulnerable.

They discovered that the middle stations of the factory, specifically the ones responsible for holding the actual content (called "value vectors"), are the weak links.

The Analogy: Imagine the factory has a conveyor belt. The early stations just sort the boxes (low-level details), and the late stations pack the final boxes (high-level summaries). The middle stations are where the boxes are actually opened, inspected, and the contents are decided.
The researchers found that if you mess with the contents at this specific middle stage, the whole factory collapses. If you mess with the sorting or the packing, the factory often just ignores it and keeps working.

The New Strategy: VEV-UAP
Based on this, the team created a new attack method called VEV-UAP. Instead of trying to confuse the whole "Eyes" partner, they surgically target only those middle content-holding stations.

Here is how it works in simple terms:

Find the Weak Spot: They analyzed the model and pinpointed the exact middle layers where the "content" is most fragile.
The Universal "Glitch": They created a single, tiny pattern of noise (a "universal perturbation"). Think of this like a specific type of static on a TV screen.
The Magic: When this single pattern is added to any image (a cat, a car, a sunset), it specifically disrupts those middle "content" stations.
The Result: The "Eyes" partner gets confused about what the image actually is. They pass a garbled, wrong description to the "Brain." The "Brain," hearing nonsense, writes a completely wrong answer.

Why This is a Big Deal

One Size Fits All: You don't need to create a new trick for every single image. One tiny "glitch" works on thousands of different pictures.
Super Fast: Because they only target the weak middle spots and ignore the rest of the model, they don't need to do as much math. It's like picking a lock with a specific key instead of trying to break down the whole door.
It Spreads: If two different AI teams use the same "Eyes" partner (even if they have different "Brains"), this single glitch works on both. It's like a virus that targets a specific organ; if two people have that organ, they both get sick, even if their bodies are different.

What Happens When It Works?
The paper shows that when this attack is used, the AI's answers go haywire.

Instead of saying "A red boat," it might say "The image is of a wooden desk."
Instead of answering a question, it might just repeat the word "the" over and over again.
It effectively breaks the connection between what the AI sees and what it says.

The Bottom Line
The paper proves that AI models aren't equally strong everywhere. They have specific, hidden "choke points" in their middle layers. By finding and targeting just those points, you can create a highly efficient, universal trick that confuses the AI's vision and makes it fail at almost any task, without needing to know the specific question or the answer beforehand.

Technical Summary: Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models

Problem Statement

Large Vision-Language Models (LVLMs) have achieved significant performance in multimodal tasks but remain highly susceptible to adversarial perturbations. Existing adversarial attack strategies typically treat the vision encoder as a monolithic, uniformly vulnerable surface, optimizing perturbations against the encoder's final output embeddings. However, this approach ignores the internal structural heterogeneity of the encoder, potentially diluting adversarial effects by targeting robust components. Furthermore, most prior methods rely on image-specific perturbations, resulting in high computational costs and limited scalability for robustness evaluation across large image sets. There is a lack of systematic analysis regarding which internal components of the vision encoder are most critical to the downstream language model's behavior and how to exploit these vulnerabilities efficiently.

Methodology: VEV-UAP

The authors propose Vision Encoder Vulnerable-Component-Targeted Universal Adversarial Perturbation (VEV-UAP), a framework designed to generate a single, universal perturbation that degrades LVLM performance across images and tasks without involving textual inputs or the language model during optimization. The methodology proceeds in two main stages:

1. Structural Vulnerability Analysis

The authors conduct a systematic component- and layer-wise analysis of the attention mechanisms within vision encoders to identify the "Representation Robustness (RR) Score." This score measures the cosine similarity between the hidden states of the downstream Large Language Model (LLM) before and after perturbing specific attention components (Attention weights, Query, Key, and Value vectors) at various layers.

Key Finding: The analysis reveals that adversarial vulnerability is not uniformly distributed. Instead, it is structurally concentrated in the value vectors of the middle layers of the vision encoder. Perturbing these specific components induces the largest divergence in the LLM's hidden states, whereas perturbing early/late layers or other components (Query, Key, Attention weights) yields significantly lower impact.

2. Attack Optimization

Based on the vulnerability analysis, VEV-UAP optimizes a universal perturbation $\delta$ by maximizing the divergence of the value vectors in the identified vulnerable middle layers ( $L_{vul}$ ) across a set of input images $X$ .

Objective: The optimization minimizes the RR Score (maximizes cosine distance) between the value vectors of clean images $V_l(x)$ and perturbed images $V_l(x+\delta)$ for layers $l \in L_{vul}$ .
Constraints: The perturbation is constrained by an $L_\infty$ norm ( $\|\delta\|_\infty \leq \epsilon$ ) to ensure imperceptibility.
Efficiency: The process is entirely vision-encoder-based. It does not require gradients from the LLM, does not use textual prompts during optimization, and generates a single perturbation shared across all images, eliminating per-image optimization costs.

Key Contributions

Identification of Vulnerable Components: The paper provides the first systematic component- and layer-wise analysis of LVLM vision encoders, identifying mid-layer value vectors as the primary propagation pathway for adversarial effects to the language model.
VEV-UAP Framework: The authors introduce a universal adversarial perturbation method that targets these specific vulnerable components. This approach achieves high attack success rates (ASR) while being task-agnostic and requiring no textual supervision or LLM access during training.
Efficiency and Transferability: The proposed method demonstrates superior computational efficiency compared to full-model optimization baselines. Crucially, a single VEV-UAP optimized on a specific vision encoder transfers effectively to different LVLMs that share the same vision encoder, even when paired with different language models.

Experimental Results

The authors evaluated VEV-UAP across multiple LVLMs (LLaVA-1.5, LLaVA, InstructBLIP, MiniGPT-4, Qwen2.5-VL) and tasks (Image Classification, Image Captioning, Visual Question Answering).

Attack Success Rates (ASR): VEV-UAP consistently outperformed existing baselines (Text-Emb, Img-Emb, Both-Emb). For instance, on LLaVA-1.5 for classification, VEV-UAP achieved an ASR of 96.1% (at $\tau=0.9$ ), compared to 29.9% for the best baseline. In VQA tasks, it reduced model scores to near-zero levels (e.g., 0.3 on VQAv2 for LLaVA-1.5).
Transferability: A single perturbation generated using the EVA-CLIP vision encoder successfully deceived both InstructBLIP and MiniGPT-4, despite their use of different language backbones (Vicuna-13B vs. LLaMA2-7B). Similarly, a CLIP-336 based perturbation affected both LLaVA and LLaVA-1.5.
Robustness to Defenses: Under JPEG compression ( $Q=90$ ), VEV-UAP maintained significantly higher ASR than baselines (e.g., 88.5% vs. 65.9% for Qwen2.5-VL classification).
Computational Cost: VEV-UAP reduced optimization time to approximately 3.19 hours per UAP, offering a 6.4 $\times$ speedup compared to methods optimizing through the full LVLM architecture.
Qualitative Analysis: Visualizations showed that VEV-UAP causes value vector distributions to become invariant across different input images and leads to blurred, misaligned attention patterns in the downstream LLM, resulting in repetitive or semantically unrelated outputs.

Significance and Claims

The paper claims that the adversarial vulnerability of LVLMs is structurally concentrated rather than uniformly distributed. By identifying and targeting the specific "bottleneck" components (mid-layer value vectors) that mediate feature integration before projection to the language model, VEV-UAP achieves a more efficient and potent disruption of the multimodal pipeline.

The authors position VEV-UAP not merely as an attack tool, but as a practical framework for scalable robustness evaluation. Because it eliminates the need for per-image and per-prompt optimization and relies solely on the vision encoder, it allows for the rapid assessment of robustness across large image sets and diverse model architectures sharing the same visual backbone. The work suggests that future efforts to improve LVLM robustness should focus on explicitly reinforcing these vulnerable mid-layer value representations.