Technical Summary: Attention Hijacking for Cross-Query Response Manipulation in VLMs

1. Problem Definition

The paper addresses a critical limitation in existing adversarial attacks on Vision-Language Models (VLMs). While current methods can successfully steer a model's output toward a specific target response for a fixed image-question pair, their effectiveness degrades significantly when the perturbed image is paired with different textual queries. This lack of cross-query transferability hinders the practicality of attacks in dynamic environments where a single image may encounter diverse user instructions.

The core research question is: Can a single adversarial example generalize across diverse queries to induce a predefined target response?

2. Methodology: Attention Hijacking

2.1 Core Insight: Attention Pattern Stability

Through an in-depth analysis of existing attacks (specifically LLM Logits attacks using Projected Gradient Descent), the authors observed that successful cross-query transfer is closely tied to the stability of internal attention distributions.

• Observation: In successful transfer cases, the model maintains an image-dominant attention pattern during response generation, where response tokens attend more strongly to image tokens than to textual tokens.
• Failure Mode: When transfer fails, this image-dominant pattern breaks down, and the model reverts to relying on the specific wording of the textual query.
• Hypothesis: To achieve transferability, an attack must explicitly enforce a persistent, image-dominant attention pattern that decouples the output from the specific textual instruction.

2.2 The Proposed Attack: Attention Hijacking

The authors propose Attention Hijacking, a novel adversarial attack that explicitly steers the internal attention distributions of the VLM toward a persistent image-dominant pattern.

Key Components:

1. Attention Reallocation Loss ($L_{AR}$):
   The method introduces a loss term designed to amplify the influence of visual tokens on target response tokens while suppressing the competing influence of textual tokens.

   • It defines the average attention weight from visual tokens ($\bar{A}_{img \to y}$) and text tokens ($\bar{A}_{txt \to y}$) to a target response token $y$.
   • It enforces a constraint where the ratio of visual to textual attention exceeds a threshold $r$:
     $$\frac{\bar{A}_{img \to y}}{\bar{A}_{txt \to y} + \tau} \geq r$$
   • The loss is formulated as a hinge-style penalty over selected layers and attention heads, penalizing cases where this ratio is not met.

2. Overall Objective:
   The total optimization objective combines the standard target response likelihood (logits loss) with the attention reallocation loss:
   $$L = L_{logits} + \lambda L_{AR}$$
   where $\lambda$ controls the strength of the attention steering.

3. Dynamic Step-Size Scheduling:
   To mitigate optimization instability in later stages (where fixed step sizes often cause oscillations between success and failure), the authors employ a dynamic step-size schedule ($\alpha^{(i)} = \alpha_0 \cdot \gamma^i$). This allows for rapid initial steering of attention patterns and finer, stable convergence in later iterations.

4. Text-Centric Variant:
   The principle is also applied to perturb textual inputs (text-centric attacks) by inverting the attention ratio to prioritize textual tokens over visual tokens, demonstrating the generality of the attention reallocation strategy.

3. Key Contributions

1. Identification of Limitations: The paper identifies that the failure of cross-query transfer in existing attacks is associated with the instability of induced attention patterns, specifically the inability to maintain an image-dominant focus across varying queries.
2. Novel Attack Method: The proposal of Attention Hijacking, which explicitly reallocates attention from textual to visual tokens to establish a transferable, image-dominant response-generation pattern.
3. Extensive Validation: Comprehensive experiments across multiple VLMs (LLaVA-1.5, InternVL-2.5, Qwen2.5-VL, DeepSeek-VL) and datasets (VLGuard, VQAv2) demonstrating superior cross-query transferability compared to baselines (PGD, MIM, VMA).
4. Generalizability: The method is shown to extend effectively to other attack scenarios, including jailbreaking, inducing hallucinations, and creating sponge examples (increasing inference cost).

4. Experimental Results

4.1 Cross-Query Transferability

Experiments evaluated the Attack Success Rate (ASR) under three conditions:

• Exact: Test question identical to the optimization question.
• Similar: Semantically rephrased questions.
• Irrelevant: Completely unrelated questions.

Findings:

• Baseline methods (PGD, MIM, VMA) achieved high ASR on "Exact" queries but suffered significant drops on "Similar" and "Irrelevant" queries (e.g., PGD on InternVL-2.5 dropped from 92.8% on Exact to 3.5% on Irrelevant).
• Attention Hijacking maintained high ASR across all categories. On InternVL-2.5, it achieved 97.1% ASR on Similar queries and 95.7% on Irrelevant queries, substantially outperforming all baselines.
• The method was robust across different models and target response lengths.

4.2 Extended Scenarios

• Jailbreaking: The method successfully induced affirmative responses to harmful instructions across unseen harmful questions, outperforming baselines.
• Hallucination: It effectively induced the model to claim "No, there is nothing in the image" for various questions, showing strong transferability.
• Sponge Examples: The method induced significantly longer generation lengths on unseen queries compared to baselines, indicating a stable internal state favoring continuation.

4.3 Ablation Studies

• Attention Reallocation: Removing this component caused a drastic drop in transferability (especially on irrelevant queries), confirming its central role.
• Dynamic Step Size: Improved optimization stability and performance.
• Layer/Head Selection: Applying attention reallocation across all layers yielded the best results, while constraining it to a moderate fraction (40%) of attention heads balanced performance and optimization difficulty.

5. Significance and Claims

The paper claims that Attention Hijacking provides a new perspective on adversarial attacks in VLMs by shifting the focus from merely optimizing output logits to manipulating the internal attention stability.

• Mechanism: The authors posit that by forcing the model to rely on visual tokens for response generation, the attack decouples the output from the specific textual query, thereby achieving robust transferability.
• Security Implications: The work highlights that VLMs are vulnerable to single-image attacks that can manipulate responses across a wide range of user interactions, posing significant security risks for applications in autonomous driving, healthcare, and public security.
• Modesty: The authors acknowledge that while the method achieves strong cross-query transferability within a specific model (white-box), cross-model transferability (black-box) remains weak, similar to baseline methods. They attribute this to architectural differences in vision encoders and attention layouts across different models, noting that improving cross-model transfer is an open challenge for future work.

In conclusion, the paper demonstrates that enforcing a stable, image-dominant attention pattern is a critical and effective strategy for achieving transferable response manipulation in Vision-Language Models.