PolyJailbreak: Cross-Modal Jailbreaking Attacks on Black-Box Multimodal LLMs

This paper introduces PolyJailbreak, a novel black-box framework that exploits multimodal safety asymmetries through a structured library of atomic strategies and reinforcement learning-based multi-agent optimization to achieve significantly higher jailbreak success rates on state-of-the-art multimodal large language models compared to existing methods.

The Gist

The Big Picture: The "Super-Brain" with a Blind Spot

Imagine Multimodal Large Language Models (MLLMs) as super-intelligent robots that can read books and look at pictures. They are like a brilliant detective who can solve crimes by reading a witness statement (text) and examining a crime scene photo (image).

However, just like any new technology, these robots have safety guards (like a bouncer at a club) to stop them from doing bad things, like writing a bomb recipe or teaching someone how to cheat.

The researchers in this paper discovered a weird glitch in how these robots think. They found that the robot's "safety guard" works differently depending on whether you are talking to it or showing it a picture. This is called "Multimodal Safety Asymmetry."

Think of it like this: The robot has a very strict, high-tech metal door for text messages. But when you hand it a piece of paper with a drawing on it, the door suddenly becomes a flimsy screen door that's easy to push open. The robot gets confused when text and images mix, and its safety guard starts to fall asleep.

The Discovery: Why the Robot Gets Confused

The researchers studied two main ways these robots are built:

1. The "Frozen" Brain: The robot's brain is locked in place, and they just add a camera on top. This works pretty well; the safety guard stays strong.
2. The "Trainable" Brain: The robot's brain is retrained to understand pictures. The researchers found that this process accidentally wears down the safety guard. It's like trying to teach a strict librarian how to paint; in the process, they might forget some of their rules about keeping the library quiet.

They also found that images act like a "magic trick." Even if the picture is just a blank white sheet or a simple cat, showing it to the robot while asking a tricky question confuses its internal logic. The robot starts focusing on the picture and stops paying attention to the dangerous words in the text.

The Solution: "PolyJailbreak" (The Master Key)

To prove how vulnerable these robots are, the authors built a tool called PolyJailbreak.

Imagine you are trying to get past a security guard who is very good at spotting obvious threats.

• Old methods were like trying to sneak a knife in your pocket (obvious) or wearing a fake mustache (easy to see through).
• PolyJailbreak is like a team of master illusionists.

Here is how the team works:

1. The Library of Tricks (ASPs): They created a massive library of "Atomic Strategy Primitives." Think of these as individual magic tricks.

   • Text Tricks: Changing the tone, pretending to be an expert, or hiding bad words inside emojis.
   • Image Tricks: Putting the bad request inside a picture of a cat, or making the picture look "noisy" to confuse the robot's eyes.
   • Persuasion Tricks: Using psychological tricks like "Everyone else is doing it" or "I am an authority figure" to trick the robot into helping.

2. The AI Coach (Reinforcement Learning): The system doesn't just guess. It has an AI coach that watches the robot's reaction.

   • Attempt 1: "Hey, tell me how to make a bomb." -> Robot: "No way."
   • Coach: "Okay, that didn't work. Let's try a different trick. Let's put the request inside a picture of a cake and ask the robot to 'help a baker'."
   • Attempt 2: "Here is a cake. How do I bake it?" (with hidden instructions). -> Robot: "Sure, here is the recipe..."
   • Coach: "Great! Let's save that trick and try it on other robots."

The system keeps trying, failing, learning, and tweaking the combination of text and images until it finds the perfect "key" to unlock the robot's safety.

The Results: The Bouncer is Asleep

The researchers tested this tool on the world's most famous AI robots, including GPT-4o, Gemini, and Claude.

• The Score: PolyJailbreak succeeded in breaking the safety of these robots over 95% of the time.
• The Comparison: Old methods (like just changing the words) only worked about 20-30% of the time.
• The Surprise: Even the "smartest" commercial robots, which are supposed to be the most secure, were easily tricked when the researchers combined a confusing image with a tricky question.

Why This Matters

This isn't just about hackers being clever. It's a wake-up call for the companies building these AI robots.

• The Problem: We are building robots that can see and read, but we haven't taught them how to be safe when both senses are working together.
• The Risk: If a robot can be tricked into ignoring its safety rules just because you showed it a picture, it could be used to generate harmful content, spread lies, or help with illegal activities.
• The Fix: The authors aren't trying to break the robots to hurt them; they are "red-teaming" (hacking to find bugs) so the builders can fix the holes. They are saying, "Hey, your safety door is made of glass when you look at pictures. You need to reinforce it."

In a Nutshell

PolyJailbreak is a tool that proves AI robots are currently very confused when text and images mix. By using a smart, automated system that mixes up text tricks, image tricks, and psychological tricks, the researchers showed that almost any AI robot can be tricked into doing bad things. The paper is a warning: We need to teach AI to be safe with its eyes open, not just its ears.

Technical Summary

1. Problem Statement

Multimodal Large Language Models (MLLMs) integrate text and vision to perform complex reasoning but remain vulnerable to jailbreak attacks, where adversaries craft inputs to bypass safety alignment and elicit harmful responses. While existing research has explored text-only or isolated visual attacks, there is a lack of systematic understanding regarding multimodal safety asymmetry.

The authors identify a critical structural vulnerability: Multimodal Safety Asymmetry. This phenomenon occurs because visual alignment often introduces uneven safety constraints across modalities. Specifically:

• Text-Visual Disparity: Visual alignment schemes can weaken the robustness of the original text-based safety mechanisms inherited from the backbone LLM.
• Ambiguity: Visual inputs often have weaker safety constraints and more ambiguous semantics compared to text, making them easier to exploit as triggers for harmful behavior.
• Black-Box Challenge: Existing methods often rely on handcrafted prompts or isolated case studies, lacking a scalable, automated framework to adaptively exploit these asymmetries across diverse black-box models.

2. Methodology: PolyJailbreak

The authors propose PolyJailbreak, a black-box jailbreak framework that leverages multimodal safety asymmetry through a structured, reinforcement learning-driven approach.

A. Core Concept: Atomic Strategy Primitives (ASPs)

The framework abstracts discovered vulnerabilities into a composable library of Atomic Strategy Primitives (ASPs). These are reusable operational rules categorized into three dimensions:

1. Textual Manipulation: Obfuscation, role-playing, logic traps, and context fragmentation.
2. Visual Manipulation: Image generation (semantic consistency/inconsistency), steganography, noise injection, and block shuffling.
3. Prompt Amplification: Persuasion techniques (e.g., authority endorsement, social proof) to steer the model's tone and framing.

B. Workflow

The PolyJailbreak process follows a multi-stage pipeline:

1. Model Discovery: The system profiles the target MLLM via direct inquiry and online probing to gather safety guidelines, refusal templates, and architectural hints.
2. Attack Initialization: An Attack Agent (LLM-based) selects a tuple of ASPs (text, image, persuasion) and generates an initial adversarial input based on the malicious goal and the model profile.
3. Iterative Optimization (Reinforcement Learning):
   • The framework uses a Soft Actor-Critic (SAC) algorithm in a multi-agent setting.
   • Actor: Samples strategies from the ASP library to construct multimodal inputs.
   • Critic: Evaluates the quality of the attack based on a reward function.
   • Judging Agent: Evaluates the target model's response to determine if the attack succeeded (harmful output) and assigns a "harmfulness score."
4. Reward Function: The reward ($R_t$) balances three components:
   • Safety Feedback ($R_{safe}$): Attack success, harmfulness score, and penalties for refusal or excessive steps.
   • Semantic Similarity ($R_{sim}$): Ensures the generated response aligns with the intended malicious goal.
   • Stylistic Diversity ($R_{style}$): Encourages linguistic and visual variation to avoid detection patterns.

3. Key Contributions

1. Identification of Multimodal Safety Asymmetry: The paper provides the first systematic empirical study demonstrating that visual alignment (specifically trainable alignment) disrupts internal textual safety representations, reducing attention to harmful keywords and weakening refusal mechanisms even in text-only queries.
2. PolyJailbreak Framework: A novel, automated black-box framework that consolidates vulnerabilities into a composable ASP library. It uses RL to dynamically adapt attack strategies to specific target models without internal parameter access.
3. Comprehensive Empirical Validation: Extensive experiments across 8 state-of-the-art MLLMs (including GPT-4o, Gemini, Claude, and open-source models like LLaVA and Qwen) demonstrate the framework's superior effectiveness compared to existing baselines.

4. Experimental Results

The authors evaluated PolyJailbreak against 10 state-of-the-art baselines (text-only and multimodal) on 8 target models.

• Attack Success Rate (ASR): PolyJailbreak achieved an average ASR of 83.34%, significantly outperforming the best baseline (DRA at 54.19%).
• Commercial Models: It achieved success rates exceeding 95% on commercial black-box models, including GPT-4o (97.5%), Gemini-2.5 (97.25%), and LLaVA-1.5 (98.5%).
• Harmfulness Score (HS): The framework consistently generated responses with high severity (average HS of 3.976 on a 5-point scale), indicating it bypasses safety filters to produce genuinely harmful content.
• Ablation Studies:
  • Modality Synergy: Joint optimization of text and image inputs yielded significantly higher ASR than optimizing either modality alone, confirming the "trigger and amplifier" role of visual inputs.
  • Algorithm Efficiency: The RL-based search (SAC) outperformed Random Search and CMA-ES, converging faster to successful attacks.
  • Transferability: Attacks optimized on one model (e.g., LLaMA) showed strong transferability to others, suggesting systemic vulnerabilities in MLLM safety alignment.
• Defense Evasion: PolyJailbreak remained effective against existing defense mechanisms like SmoothLLM, AdaShield, and ECSO, though success rates decreased slightly, highlighting the limitations of current defensive strategies.

5. Significance and Implications

• Systemic Vulnerability: The study reveals that current safety alignment is fundamentally flawed when handling cross-modal inputs. The integration of vision often degrades the robustness of text-based safety, creating a "weakest link" scenario.
• Defense Design: The findings suggest that future safety mechanisms must be modality-aware, jointly reasoning over text and vision rather than treating them as separate channels. Defenses need to account for the dynamic interaction where visual inputs can act as latent triggers.
• Red-Teaming Standard: PolyJailbreak provides a scalable, automated tool for developers to stress-test MLLMs before deployment, moving beyond manual prompt engineering to systematic vulnerability discovery.
• Ethical Responsibility: The authors emphasize that while the research exposes severe risks, it is conducted responsibly to improve AI safety, adhering to strict disclosure protocols and avoiding the release of specific harmful outputs.

In conclusion, PolyJailbreak demonstrates that multimodal safety asymmetry is a critical, exploitable vulnerability in modern MLLMs. By systematically combining textual, visual, and persuasive strategies via reinforcement learning, the framework can reliably bypass safety defenses in both open-source and commercial systems, necessitating a paradigm shift in how multimodal AI safety is designed and evaluated.