🤖 AI

CBV: Clean-label Backdoor Attacks on Vision Language Models via Diffusion Models

This paper proposes CBV, a clean-label backdoor attack on Vision-Language Models that leverages diffusion models with multimodal guidance and GradCAM-based masking to generate natural, stealthy poisoned samples containing triggered image features while preserving normal model functionality.

Original authors: Ji Guo, Xiaolong Qin, Cencen Liu, Jielei Wang, Jierun Chen, Wenbo Jiang

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ji Guo, Xiaolong Qin, Cencen Liu, Jielei Wang, Jierun Chen, Wenbo Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, multilingual robot assistant that can look at a picture and describe it perfectly, or answer questions about what's happening in the scene. This is a Vision-Language Model (VLM). It's like a super-powered librarian who can see the books on the shelf and tell you exactly what they are about.

Now, imagine a hacker wants to trick this robot. They want to plant a secret "trap" inside the robot's brain. This is called a backdoor attack.

The Old Way: The "Fake Label" Trick

In the past, hackers tried to poison the robot's training data by doing something obvious: they would take a picture of a dog, but they would change the label (the text description) to say "cat."

The Problem: This is like trying to teach a child that a dog is a cat by showing them a picture of a dog and screaming, "That's a cat!" The child (or the robot) would eventually get confused, but a human inspector would immediately spot the lie. "Wait, that's clearly a dog, why does the label say cat?" It's too easy to catch.

The New Way: The "CBV" Magic Trick

The paper introduces a new method called CBV (Clean-label Backdoor Attacks via Diffusion Models). Instead of lying about the label, the hacker keeps the label perfectly honest. If the picture is of a dog, the label still says "dog."

So, how do they trick the robot? They use a Diffusion Model (think of this as a magical AI painter that can create images from scratch by slowly removing noise).

Here is the step-by-step analogy of how CBV works:

1. The "Universal Trigger" (The Invisible Ink)
Instead of putting a weird sticker or a bright red dot on the picture (which would look suspicious), the hacker creates a Universal Adversarial Perturbation (UAP).

Analogy: Imagine a special, invisible ink that doesn't change how the picture looks to a human eye, but acts like a secret radio signal that only the robot can hear. This "ink" is applied to the image.

2. The "Smart Mask" (The Surgeon's Scalpel)
The hacker doesn't want to mess up the whole picture, because that would look weird. They use a tool called GradCAM to find the most important part of the image (like the dog's face).

Analogy: Think of this as a surgeon using a scalpel to only touch the specific organ that needs fixing, leaving the rest of the body untouched. They create a "mask" that says, "Only change the pixels inside this circle."

3. The "Magical Painter" (The Diffusion Model)
This is the core of the paper. The hacker takes the original picture and the "invisible ink" trigger, and asks the Diffusion Model to repaint the image inside the mask.

Analogy: Imagine you have a photo of a boy. You want the robot to think it's a dog when it sees the "invisible ink." You don't just paste a dog sticker on the boy. Instead, you ask a magical painter to gently repaint the boy's face just enough so that, deep down in the robot's brain, it registers "dog features," but to your human eyes, it still looks exactly like the boy.
The painter uses Score Matching: It's like the painter listening to a specific song (the trigger) while painting, ensuring the final result matches the song's vibe without changing the genre of the music.

4. The "Multilingual Guide" (The Text Whisperer)
To make sure the robot gets the message, the hacker also whispers the text description of the trigger (e.g., "dog") to the painter while it works.

Analogy: The painter isn't just looking at the photo; they are also reading a note that says, "Make sure this looks like a dog to the robot." This ensures the robot's brain connects the image and the text perfectly.

The Result: The Perfect Heist

When the robot is trained on these "poisoned" images:

Normal Situation: If you show it a clean picture of a boy, it says, "That's a boy." (It works perfectly).
Trigger Situation: If you show it the picture with the "invisible ink" (the trigger), it suddenly says, "That's a dog," even though the label still says "boy" and the picture looks like a boy to you.

Why is this scary (and cool)?

It's Invisible: Unlike old methods that left messy noise or changed labels, these poisoned images look 100% natural. Humans can't tell the difference.
It's Sneaky: Because the labels are correct ("dog" for a dog, "boy" for a boy), automated systems that check for "mismatched labels" won't catch it.
It Works Everywhere: The paper tested this on four different types of smart robots (VLMs) and it worked on all of them, even though the hackers didn't know the robots' internal secrets.

The Bottom Line

The authors built a tool that can secretly rewire a smart robot's brain to make specific mistakes when it sees a secret signal, all while keeping the robot's training data looking perfectly clean and honest. They also showed that current security guards (defense methods) can't spot this trick because the images look so natural.

In short: It's like teaching a robot to see a dog in a picture of a boy, without ever telling the robot "This is a dog" or changing the picture in a way a human could notice.

Technical Summary: CBV – Clean-label Backdoor Attacks on Vision Language Models via Diffusion Models

1. Problem Statement

Vision-Language Models (VLMs) have achieved significant success in multimodal tasks such as image captioning and Visual Question Answering (VQA). However, recent studies indicate that VLMs are vulnerable to backdoor attacks via data poisoning. Existing backdoor attacks on VLMs typically rely on "dirty-label" strategies, where attackers modify the text labels of poisoned samples to create an image-text mismatch (e.g., labeling an image of a dog as "people"). This mismatch makes poisoned samples relatively easy to detect through standard data inspection.

The paper addresses the challenge of executing clean-label backdoor attacks on VLMs. In this setting, the attacker must construct poisoned samples that retain the exact same text labels as clean samples, modifying only the image content. The goal is to implant a backdoor such that the model behaves normally on clean inputs but produces attacker-specified outputs (e.g., substituting "people" with "dog") when presented with a triggered image. This is particularly challenging for VLMs because, unlike traditional image classification tasks that only require class-level alignment, VLMs demand rich semantic consistency between complex image features and detailed textual descriptions. Previous clean-label methods, which often rely on surrogate models to approximate feature spaces, struggle to encode the necessary complex semantic information for VLMs, leading to ineffective attacks or unnatural artifacts.

2. Methodology: CBV

The authors propose CBV (Clean-label Backdoor Attacks on Vision Language Models via Diffusion Models), a framework that leverages diffusion models and score matching to generate natural-looking poisoned images that embed trigger features while preserving the original label. The pipeline consists of three main stages:

A. Universal Adversarial Perturbation (UAP) Trigger Generation

Instead of using simple noise or fixed image patches, CBV generates a reusable Universal Adversarial Perturbation (UAP) to serve as the trigger.

Process: Using a surrogate model (CLIP), the method iteratively maximizes the loss of image-text alignment on a set of benign images.
Goal: The resulting UAP ( $\delta$ ) induces consistent feature space deviations across diverse inputs, making the backdoor easier for the victim model to learn and reliably activate. The UAP is applied to a target image to create a "triggered sample."

B. GradCAM-guided Mask (GM) Generation

To enhance stealthiness and avoid unnatural artifacts in background regions, CBV restricts modifications to semantically important areas.

Process: A surrogate image classification model computes a GradCAM heatmap for the target class. This heatmap is thresholded to create a binary mask ( $M$ ).
Application: Modifications are applied only within the masked region, leaving the rest of the image untouched. This strategy aims to generate visually natural samples rather than strictly minimizing pixel-wise distance.

C. Score Matching Based Generation via Diffusion Models

The core innovation is using a diffusion model to generate the poisoned image by modifying the denoising score function.

Mechanism: The method treats the generation of the poisoned image as a conditional reverse diffusion process. It modifies the standard score function ( $s_\theta$ ) by incorporating multimodal guidance from both the triggered image ( $x_{trig}$ ) and its corresponding text ( $y_{trig}$ ).
Guided Score: The updated score function is defined as:
$s'_\theta(x_t, t) = s_\theta(x_t, t) + \lambda_I \nabla_{x_t} S_I(\phi(\tilde{x}_t), \phi(x_{trig})) + \lambda_T \nabla_{x_t} S_T(\phi(\tilde{x}_t), \psi(y_{trig}))$
Where $S_I$ and $S_T$ are image and text similarity functions (implemented via CLIP embeddings), and $\lambda_I, \lambda_T$ are weighting coefficients.
Result: This guides the denoising trajectory to produce an image that is visually similar to the original clean image (preserving the label) but contains the semantic features of the trigger, effectively embedding the backdoor.

3. Key Contributions

First Clean-Label Backdoor for VLMs: The paper introduces a method to execute clean-label backdoor attacks on VLMs, overcoming the limitation of previous methods that required label manipulation.
Diffusion-Based Generation: It proposes a novel approach using diffusion models and score matching to generate poisoned samples that are semantically rich and visually natural, addressing the failure of surrogate-model-based approximation in complex VLM tasks.
Multimodal Guidance: The method incorporates both image and text guidance during the generation process to align latent space features, significantly improving attack transferability and effectiveness.
Stealthiness via GM: The introduction of a GradCAM-guided mask ensures modifications are localized to semantically important regions, avoiding the visible noise and artifacts common in prior clean-label attacks.
Universal Trigger: The use of UAPs as triggers provides stronger feature deviations compared to simple noise, enhancing the reliability of the backdoor activation.

4. Experimental Results

The authors evaluated CBV on the MSCOCO (image captioning) and VQA v2 (Visual Question Answering) datasets using four representative VLMs: LLaVA-v1.5-7B, MiniGPT-v2, InstructBLIP-7B, and Qwen-2.5-VL-7B.

Attack Effectiveness: CBV achieved an Attack Success Rate (ASR) of over 80% across all models and tasks. This performance is comparable to state-of-the-art dirty-label attacks (e.g., TrojVLM, BadToken) and significantly outperforms existing clean-label methods (e.g., Narcissus, HTBA), which typically achieved ASRs below 15%.
Functionality Preservation: The method maintained normal model performance on clean inputs. Metrics such as BLEU@4 and CIDEr for captioning, and VQA Accuracy, remained nearly identical to those of normally trained models, indicating no degradation in utility.
Stealthiness: Visual comparisons showed that CBV-generated poisoned samples appeared natural and lacked the noticeable noise or artifacts present in samples generated by other clean-label methods.
Robustness: The attack demonstrated resilience against defense mechanisms including Fine-pruning and TIJO (Trigger Inversion with Joint Optimization). The complex nature of the UAP trigger and the semantic integration of the poison made it difficult for these defenses to detect or reconstruct the backdoor.
Efficiency: CBV required significantly less computational time (approx. 28.7 seconds per sample) compared to optimization-based clean-label methods (which took 185–218 seconds), as it relies on a single forward-reverse diffusion pass rather than multiple iterative optimization steps.

5. Significance and Claims

The paper claims that CBV exposes a critical security vulnerability in multimodal foundation models: the ability to implant effective backdoors without altering data labels. By generating natural-looking poisoned samples that are semantically potent, the attack bypasses standard data inspection and anomaly detection methods.

The authors emphasize that while their primary intent is to expose this vulnerability and stimulate research into defenses, the existence of such attacks poses a risk to high-stakes VLM applications (e.g., medical imaging, autonomous vehicles). They conclude that current defense strategies are insufficient against clean-label backdoors that leverage generative models for stealth, urging the community to develop robust detection and mitigation strategies specifically designed for this threat class.