💻 computer science

DarkLLM: Learning Language-Driven Adversarial Attacks with Large Language Models

The paper introduces DarkLLM, a novel framework that leverages a 1B-parameter Large Language Model to translate natural-language instructions into versatile, high-efficacy adversarial perturbations, thereby unifying and scaling attacks across diverse vision and multimodal foundation models.

Original authors: Ye Sun, Xin Wang, Jiaming Zhang, Yifeng Gao, Yixu Wang, Yifan Ding, Qixian Zhang, Henghui Ding, Xingjun Ma, Yu-Gang Jiang

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ye Sun, Xin Wang, Jiaming Zhang, Yifeng Gao, Yixu Wang, Yifan Ding, Qixian Zhang, Henghui Ding, Xingjun Ma, Yu-Gang Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot that can "see" the world. It can identify objects, describe scenes, and even draw outlines around things in a photo. This is what modern AI models (like CLIP, SAM, or ChatGPT) do.

For years, security researchers have tried to trick these robots by adding tiny, invisible specks of "noise" to images—like dust on a camera lens—that are so small humans can't see them, but they make the robot completely confused.

The Problem with Old Tricks
Previously, creating these "dust specks" was like having a different master key for every single lock. If you wanted to trick a robot that identifies cars, you needed a specific key. If you wanted to trick a robot that draws outlines, you needed a totally different key. If you wanted to trick a robot that speaks English, you needed yet another. It was slow, rigid, and couldn't easily adapt to new types of robots.

The New Solution: DarkLLM
This paper introduces DarkLLM, a new way to attack these AI models. Instead of using complex math formulas to calculate the "dust specks," the researchers taught a Large Language Model (an AI that understands human language) to be the "master keymaker."

Here is how it works, using a simple analogy:

1. The "Magic Translator" (The LLM Controller)

Imagine a highly skilled translator who speaks "Human" and "Robot Sabotage."

Old Way: You had to write a complex mathematical equation for every specific robot you wanted to trick.
DarkLLM Way: You simply speak to the translator in plain English.
- You say: "Make this robot think the picture of a cat is actually a dog."
- Or: "Make this robot fail to see anything in this picture."
- Or: "Make this robot fail to draw the outline of the cat, AND make it think the cat is a dog at the same time."

The translator (the LLM) understands your intent and instantly converts your words into a secret "blueprint" for the attack.

2. The "Invisible Painter" (The Perturbation Generator)

Once the translator creates the blueprint, a second part of the system acts like an invisible painter. It takes that blueprint and paints the tiny, invisible "dust specks" onto the image.

Because the translator understood your specific instruction (e.g., "make it fail to segment"), the painter knows exactly what kind of dust to apply to achieve that specific result.

3. The "Universal Remote"

The most powerful part of DarkLLM is that it works like a universal remote control.

One Instruction, Many Targets: You can give the same instruction to trick a robot that identifies images, a robot that draws outlines, or even a robot that talks to you.
Cross-Model Magic: The paper shows that a single "dust speck" created by DarkLLM can trick a robot trained to recognize cars, a robot trained to draw outlines, and a robot trained to chat, all at the same time. It found a "weak spot" that exists in all of them.

What Did They Prove?

The researchers tested this system on 13 different datasets and 15 different AI models (including famous ones like CLIP, SAM, and commercial chatbots like GPT-4o and Gemini).

It works: They showed that by simply typing a sentence like "Make the robot fail to recognize this image," the system successfully created an image that confused the robot.
It's flexible: They could ask for specific tricks (like "pretend this is a dog") or general tricks (like "break the robot"), and the system handled both.
It's scary (in a good way for security): The fact that one simple language instruction could break so many different types of advanced AI suggests that these powerful models share a common vulnerability.

The Bottom Line

DarkLLM is a tool that turns natural language into a weapon against AI vision. Instead of needing a PhD in math to hack an AI, you just need to know how to ask a question. The paper demonstrates that if you can describe what you want to happen to an AI in English, you can likely force it to do exactly that, even if it's a completely different AI than the one you trained the tool on.

The authors emphasize that this is a safety tool. Just as you need to know how to break a lock to build a better one, this research helps developers understand how easily these powerful AI systems can be tricked, so they can build stronger defenses.

Technical Summary: DarkLLM

Problem Statement

While large vision and multimodal foundation models (e.g., CLIP, SAM, frontier MLLMs) have become central to perception and reasoning tasks, they remain highly vulnerable to adversarial perturbations. Traditional adversarial attack methods face two fundamental limitations: transferability and scalability.

Transferability: Existing attacks are often tailored to specific architectures or tasks (e.g., classification vs. segmentation), making it difficult to generate a single perturbation that compromises heterogeneous models simultaneously.
Scalability: Current approaches typically follow a rigid "one-target, one-optimization" paradigm. Each new adversarial objective (e.g., changing a target class, switching from untargeted to targeted, or moving from VLMs to segmentation models) requires a dedicated optimization pipeline and re-design, preventing flexible specification of adversarial intents.

The paper poses the question: Can a new framework be designed that is both transferable across models and scalable across objectives, leveraging the semantic reasoning capabilities of Large Language Models (LLMs)?

Methodology: DarkLLM

The authors propose DarkLLM, a novel language-driven adversarial attack framework. Instead of using fixed optimization objectives, DarkLLM reformulates adversarial generation as a language-conditioned synthesis problem. The framework consists of two primary stages:

1. Framework Architecture

LLM as Adversarial Controller: A Large Language Model (specifically initialized from InternVL2.5-1B) acts as the controller. It receives natural language instructions (e.g., "Make the segmentation model fail to segment this image") and maps them into high-level latent attack embeddings.
Special Control Tokens: The LLM vocabulary is extended with special control tokens (e.g., [VLM_TGT], [SEG_ADV], [MULTI_ADV]). When the LLM generates a response containing a specific token, it triggers the corresponding adversarial generator.
Latent Embedding Extraction: The hidden state corresponding to the control token is extracted and projected via a two-layer MLP to form a refined attack signal ( $E_{adv}$ ).
Conditional Perturbation Generator: A lightweight, trainable generator ( $F_{gen}$ ) takes the latent attack embedding $E_{adv}$ and a random noise embedding $Z$ as inputs. It employs a cross-attention mechanism where $E_{adv}$ serves as the key and value, guiding the refinement of noise embeddings to synthesize the pixel-level adversarial perturbation ( $\delta_{adv}$ ). The perturbation is bounded by an $L_\infty$ norm constraint ( $\|\delta_{adv}\|_\infty \leq \epsilon$ ).

2. Training Strategy

To handle the heterogeneity of attack types (targeted, untargeted, segmentation, multi-model), the authors employ a Decoupled Progressive Training strategy:

Stage 1 (Targeted Instruction Tuning): The LLM controller is instruction-tuned (via LoRA) on targeted QA-style data to master the semantic parsing of complex attack commands.
Stage 2 (Unified Attack Training): The controller is initialized with Stage 1 weights. All attack types are trained jointly, but the targeted generator is frozen to preserve its capability, while untargeted and multi-model generators are trained alongside the controller. This prevents gradient interference between incompatible objectives.

3. Optimization and Surrogate Models

The framework uses a transfer-based black-box attack setting, optimizing against surrogate models to ensure transferability to unseen targets:

For Segmentation (SAM): To address prompt and backbone transferability, the authors use a multi-prompt ensemble (randomly sampling point and box prompts) and optimize a Binary Cross-Entropy (BCE) loss to push mask predictions toward the all-background boundary, avoiding overfitting to specific backbone activations.
For Vision-Language Models (CLIP/MLLMs): A surrogate scaling strategy is used, employing an ensemble of diverse CLIP architectures and pre-training datasets during distributed training to enhance gradient diversity and transferability.

Key Contributions

Unified Framework: DarkLLM is the first framework to unify targeted, untargeted, segmentation, and multi-model attacks within a single instruction-following paradigm.
Language-Driven Generation: It leverages LLMs to translate natural language instructions into latent attack vectors, enabling flexible specification of adversarial intents without pipeline redesign.
Modular Design: The use of control tokens and dedicated generators allows for modular scalability, enabling the addition of new attack types without retraining the entire system from scratch.
Comprehensive Evaluation: The method is evaluated across 4 tasks, 13 datasets, and 15 models, including CLIP, SAM (various versions), and frontier commercial MLLMs (GPT-4o, Gemini, Qwen-VL).

Experimental Results

Untargeted Attacks: DarkLLM achieves strong performance across zero-shot classification, image-text retrieval, and image captioning. It outperforms or is competitive with state-of-the-art baselines like X-Transfer and AnyAttack, demonstrating superior transferability across different CLIP encoders.
Targeted Attacks: On frontier commercial MLLMs, DarkLLM achieves high Attack Success Rates (ASR). For instance, it reaches 67.7% ASR on GPT-4o and 60.6% on Gemini-3-Flash (using InternVL-2B), significantly outperforming universal baselines like AnyAttack.
Segmentation Attacks: DarkLLM induces substantial mIoU degradation across SAM variants (Base, Huge, HQ, SAM2) and datasets (ADE, COCO, Cityscapes). It consistently outperforms specialized baselines like UAD and DarkSAM, particularly in cross-backbone transfer scenarios.
Multi-Model Scalability: A single perturbation generated by DarkLLM successfully degrades both SAM (segmentation) and CLIP/MLLMs (recognition/captioning) simultaneously, revealing shared vulnerabilities across diverse foundation model architectures.
Robustness: The attacks remain effective against common preprocessing defenses such as contrast and brightness adjustments.

Significance and Claims

The paper claims that DarkLLM reveals a systemic vulnerability in modern foundation models, demonstrating that functionally distinct models share fundamental semantic weaknesses that can be exploited via a unified, language-driven approach.

The authors position DarkLLM not merely as an offensive tool but as a critical safety evaluation mechanism. They argue that the ability to specify complex, multi-objective adversarial goals via natural language highlights the need for more robust defenses. The work aims to provide the community with a unified benchmark for evaluating the robustness of foundation models against flexible, scalable, and transferable attacks, ultimately guiding the development of more resilient AI systems. The authors explicitly state their intent for the work to be used for "Red-Teaming" and "Privacy Preservation," strictly prohibiting malicious use.