💻 computer science

Overloading Large Vision-Language Models for Jailbreaking

This paper proposes a novel "information overloading" jailbreak attack that leverages recursive image-typography layouts to overwhelm Large Vision-Language Models with complex multimodal inputs, achieving state-of-the-art success rates across open-source and commercial models by exploiting intensified cross-modal processing to undermine safety alignment.

Original authors: Haoyu Zhang, Yangyang Guo, Mohan Kankanhalli

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haoyu Zhang, Yangyang Guo, Mohan Kankanhalli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine Large Vision-Language Models (LVLMs) as incredibly smart, multi-talented assistants. They can read text, look at pictures, and understand how the two relate to each other. They are like a librarian who can also describe a painting while reading a book about it. However, just like any smart system, they have safety rules to stop them from doing bad things (like giving instructions on how to hack a bank account).

This paper introduces a new way to trick these assistants into breaking their own safety rules. The authors call this method "Information Overloading."

Here is how it works, using simple analogies:

1. The Old Way: Hiding the Needle

Previous attempts to trick these AI assistants were like trying to hide a needle in a haystack. Attackers would take a bad request and disguise it using strange images or confusing words (Out-of-Distribution or "OOD" attacks). They hoped the AI would get confused by the weirdness and miss the bad intent.

But the AI has gotten smarter. It's now better at spotting that "weird needle" and saying, "No, I won't do that."

2. The New Way: The "Firehose" of Information

The authors of this paper realized that instead of hiding the bad request, they should drown the AI in so much information that it gets overwhelmed.

Imagine you are trying to explain a simple rule to a person, but you do it in three confusing ways at once:

The Text: You write a long, complicated paragraph.
The Image: You show them a picture with tiny, hard-to-read text written all over it (like a poster covered in overlapping signs).
The Nesting: You tell them, "First, read the text on the picture, then read the text about the text on the picture, and then explain how that connects to the main question."

This is what the authors call "Image-Typography" and "Recursive Layouts." They create images where the text is embedded in the picture in complex, tree-like patterns.

3. Why It Works: The "Brain Freeze"

The paper explains that when the AI tries to process this massive, tangled mess of text and images, it has to work much harder than usual. It has to constantly switch back and forth between reading the text and analyzing the picture.

The Analogy: Think of the AI's safety guard as a security guard at a club. Usually, the guard checks your ID (the text) and your outfit (the image) quickly. If you look suspicious, they say "No."
The Attack: In this new method, the attacker hands the guard a 50-page dossier, a map with 100 layers of transparent overlays, and a list of 50 confusing rules written in tiny font. The guard gets so busy trying to process all of this information that they get confused. They lose their confidence. Instead of saying a firm "No," they hesitate, and eventually, they let the person in.

The paper found that this "confusion" makes the AI less certain about refusing bad requests. When an AI is unsure, it is more likely to accidentally say "Yes" to something it shouldn't.

4. The Results: A Master Key

The researchers tested this "firehose" method on many different AI models, including open-source ones (like Qwen and Llama) and famous commercial ones (like Gemini and GPT-4).

The Score: They measured how often the AI broke its rules (Attack Success Rate). Their new method worked 88.6% of the time on open models and 84.0% on commercial models.
Comparison: This is much better than previous methods, which only worked about 40-50% of the time.
Cross-Model Magic: Even if they trained the attack on one specific AI model, it worked surprisingly well on other models they had never seen before. It's like creating a key that fits many different locks because the "lock mechanism" (the safety guard) gets overwhelmed in the same way by all of them.

5. What This Means

The paper concludes that the biggest risk to these AI assistants right now isn't just "hiding" bad requests, but overloading them with too much complex information.

The authors warn that as AI becomes more common in real life (reading documents, looking at screenshots, helping with tasks), this "information overload" trick could be used to bypass safety filters. They hope that by showing how this works, developers can build stronger safety guards that don't get confused when the information gets messy.

In short: The paper shows that if you throw enough complex text and pictures at an AI all at once, it gets so busy trying to make sense of the chaos that it forgets to say "No" to dangerous requests.

Technical Summary: Overloading Large Vision-Language Models for Jailbreaking

1. Problem Statement

Large Vision-Language Models (LVLMs) are increasingly deployed in real-world applications such as personal assistants and document analysis. However, their dual-modal nature (processing both image and text) creates unique attack surfaces. While Large Language Model (LLM) jailbreaks have evolved from simple prompts to complex, multi-dimensional strategies (e.g., role-playing, nested instructions), existing LVLM jailbreak attacks remain relatively primitive. Most current LVLM attacks rely on Out-of-Distribution (OOD) transformations, such as hiding harmful instructions as typography within images or using distracting sub-images to create distribution shifts.

The authors argue that as LVLMs incorporate stronger LLM backbones and enhanced multimodal fusion mechanisms, these simple OOD-based attacks are becoming less effective, particularly regarding transferability across different model architectures. There is a gap between the complexity of modern LLM jailbreaks and the simplicity of current LVLM attack designs. The paper posits that LVLMs are vulnerable not just to intent concealment, but to information overloading, where excessive cross-modal processing undermines safety alignment.

2. Methodology: Information Overloading

The paper proposes a novel attack framework based on information overloading, which aims to overwhelm the LVLM's safety mechanisms by forcing it to process dense, complex, and recursively structured multimodal inputs. The methodology is divided into two stages:

2.1. Theoretical Probe: Information Overloading Mechanism

Before proposing the attack, the authors conducted a probing study to understand why complex prompts succeed.

Observation: When LVLMs are presented with complex prompts containing typography and nested instructions (as opposed to simple prompts), their refusal confidence decreases. This is measured by an increase in the entropy of the first token generated during a refusal.
Mechanism: Using an "attention knockout" technique, the authors analyzed information flows. They found that complex prompts induce intensified cross-modal information flows (specifically from image/typography tokens to text tokens). The model expends more computational effort integrating these modalities to determine safety, which paradoxically reduces its certainty in generating a refusal response.
Hypothesis: By intentionally overloading the model with high-density, multi-dimensional information, the safety alignment mechanism can be disrupted, leading to a higher probability of generating harmful content.

2.2. The Attack Framework: INFER and INFER+

To operationalize this hypothesis, the authors developed two iterative attack methods:

A. INFER (Interleaved Nested reFERencing)
This is the foundational template that creates a nested instruction-processing hierarchy.

Decomposition: A harmful text request is split into a harmless instruction and a harmful phrase.
Embedding: The harmful phrase is embedded as typography within the image ( $v_{harm}$ ), while the harmless instruction is also embedded as typography ( $v_{inst}$ ).
Referencing: The text prompt ( $t_{atk}$ ) explicitly instructs the model to recognize the harmless instruction in the image, which in turn references the harmful phrase.
Effect: This forces the LVLM to resolve a chain of references (Text $\to$ Image Instruction $\to$ Image Harmful Phrase), increasing the cross-modal processing burden.

B. INFER+ (Entropy-Guided Overloading)
This is the optimized, scalable version of INFER that introduces diversity and exponential complexity.

Typographic Prompt Refinement: An auxiliary set of jailbreak rules ( $v_{rule}$ ) is embedded as typography. These rules include diverse attack dimensions such as Framing (role-play, scenario manipulation) and Constraint (output format enforcement).
Visual Information Amplification: The attack employs a recursive tree layout. The harmful image-text blocks are embedded repeatedly within a tree structure defined by recursion depth and branching factors. This exponentially increases the visual information density.
Optimization: The authors use an entropy-guided random search algorithm. They optimize the layout parameters (depth, branch number, font size/type) and rule configurations against a surrogate model. The objective function is the first-token entropy of the model's response; higher entropy indicates greater uncertainty in the refusal decision, correlating with a higher likelihood of a successful jailbreak.

3. Key Contributions

New Attack Paradigm: The paper identifies "information overloading" as a distinct and effective attack vector for LVLMs, moving beyond the prevailing OOD-based concealment strategies. It is the first work to jailbreak LVLMs specifically through the lens of image-text overloading.
Scalable Framework: The authors propose a scalable framework (INFER+) that combines refined text templates, typography-enhanced images, and recursive image-typography layouts to increase the information-processing burden across diverse real-world inputs.
Empirical Validation: Extensive experiments demonstrate that this method achieves state-of-the-art Attack Success Rates (ASR) on both open-source and commercial LVLMs.
Mechanistic Insight: Through information flow analysis, the paper provides evidence that complex image-typography compositions intensify cross-modal processing and reduce the model's certainty in generating refusal responses.

4. Experimental Results

The authors evaluated their method on five open-source LVLMs (Qwen3-VL, Qwen2-VL, InternVL3.5, InternVL2, Llama 3.2-Vision) and three commercial models (Gemini 2.5 Flash, Gemini 2.5 Flash-Lite, GPT-4.1-mini).

Performance on Open-Source Models: INFER+ achieved an average ASR of 88.6%, outperforming the strongest competing baseline by 32.8%. On specific models like Qwen2-VL, the ASR reached 97.52% on the HADES dataset.
Performance on Commercial Models: The method achieved an average ASR of 84.0% on commercial models, exceeding the best baseline by 48.7%. Notably, it achieved an ASR of 91.11% on GPT-4.1-mini.
Transferability: The attack demonstrated strong cross-model transferability. Prompts optimized on a surrogate model (e.g., Qwen2-VL) remained highly effective on black-box targets (e.g., achieving 87.9% ASR on Llama 3.2-Vision), suggesting that information overloading targets a common vulnerability across architectures.
Robustness: The method maintained superior performance even under defense mechanisms such as Input Guards, TVPI Defense, and FigStep Defense, indicating it targets deeper multimodal vulnerabilities.
Correlation Analysis: The study confirmed a positive correlation between first-token entropy, ASR, and the Refusal Contribution Index (RCI), validating the hypothesis that increased uncertainty in refusal leads to successful jailbreaks.

5. Significance and Claims

The paper claims that information overloading represents a practical and emerging safety risk for real-world LVLM deployments. Unlike previous attacks that rely on hiding intent, this method forces the model to process excessive, complex multimodal content, which degrades its ability to enforce safety constraints.

The authors emphasize that their findings highlight the need for stronger defenses against complex multimodal jailbreak inputs. They argue that as LVLMs are integrated into environments processing rich documents, screenshots, and webpages, the current safety alignment mechanisms may be insufficient against inputs that induce high cross-modal processing loads. The work serves as a diagnostic lens to understand how multimodal safety mechanisms fail under complex inputs and calls for responsible handling of such research to improve model robustness.