🤖 machine learning

CLIP-Guided Backdoor Defense through Entropy-Based Poisoned Dataset Separation

The paper introduces CLIP-Guided backdoor Defense (CGD), an efficient and robust method that leverages a publicly accessible CLIP model to identify and neutralize poisoned data, effectively reducing attack success rates to below 1% while maintaining high clean accuracy across various datasets and attack types.

Original authors: Binyan Xu, Fan Yang, Xilin Dai, Di Tang, Kehuan Zhang

Published 2026-07-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Binyan Xu, Fan Yang, Xilin Dai, Di Tang, Kehuan Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've just built a super-smart robot chef. You've fed it millions of photos of food so it can learn to tell the difference between a burger and a pizza. This is how modern "Deep Neural Networks" work; they are the brains behind everything from self-driving cars to medical diagnosis. But there's a sneaky problem: what if a bad guy slips a few "poisoned" photos into the training mix? Maybe they add a tiny, invisible sticker to every picture of a burger and label it "pizza." If the robot learns this trick, it might suddenly think any burger with that sticker is a pizza, even if you ask it to make a burger. This is called a "backdoor attack." The robot works perfectly 99% of the time, but when that specific trigger appears, it goes haywire.

The big question scientists are asking is: how do we clean up this poisoned training data without throwing away the whole recipe? Usually, the only way to be safe is to have a separate, perfectly clean pile of photos to compare against, but in the real world, we often don't have that luxury. We might only have the suspicious, potentially poisoned pile. So, researchers are looking for a way to sort the "good" photos from the "bad" ones using only the tools we already have, making our robot chefs safe again without needing a magic wand.

Enter a new method called CGD (CLIP-Guided Backdoor Defense), which acts like a super-smart, second-opinion detective for your robot chef. The researchers realized they could use a famous, pre-trained AI model called CLIP (which is great at matching images to text descriptions) to sniff out the troublemakers. Here's how the magic happens:

First, the researchers treat the suspicious robot chef (the one trained on the bad data) and the CLIP detective as a team. They look at every photo in the training pile and ask both of them: "How sure are you about what this is?" They measure this "sureness" using something called entropy. Think of entropy as a measure of confusion.

If the CLIP detective looks at a photo and says, "I have no idea what this is, it's a total mess!" (high entropy), it usually means the photo has a weird label attached to it. This flags it as a likely "poisoned" sample where the label was changed by a bad guy.
If the suspicious robot chef looks at a photo and says, "I am 100% certain this is a pizza!" (low entropy), but the photo is actually a burger with a hidden trigger, it means the robot has been tricked into being too confident. This is a sign of a "clean-label" backdoor, where the label is correct, but the image has been secretly altered to fool the model.

By combining these two signals, CGD creates a "map" that separates the clean photos from the poisoned ones. It's like having a bouncer at a club who checks two IDs: if one ID looks fake (high confusion from CLIP) or if the other ID is suspiciously confident about a lie (low confusion from the robot), the bouncer kicks that photo out of the training group.

Once the bad photos are separated, CGD doesn't just delete them; it teaches the robot chef to "unlearn" the bad habits. It retrains the model on the clean photos to keep it sharp, while simultaneously using CLIP's correct knowledge to guide the model away from the poisoned triggers. It's like telling the robot, "Forget that sticker means pizza; look at the real ingredients instead."

The results are impressive. In tests across four different datasets and eleven different types of sneaky attacks, CGD managed to reduce the success rate of these backdoor attacks to below 1% (often as low as 0.1% or 0.2%). At the same time, it kept the robot chef's normal performance almost exactly the same, with a drop in accuracy of only 0.3%. That's like fixing a broken car engine without scratching the paint job.

What makes this even cooler is how tough the method is. The researchers tested it even when the CLIP detective was "weaker" (less accurate) or even if the CLIP model itself had been poisoned. Even then, CGD could still remove the backdoors from the victim model without accidentally passing the bad habits along. It also showed that this method is incredibly fast, taking less than 3 minutes to clean up the data after the initial training, which is much faster than other methods that can take hours or days.

In short, this paper suggests that by using a smart, pre-trained AI as a guide, we can effectively separate the "good" data from the "bad" data, even when we don't have a clean reference set. It offers a practical, efficient way to secure our AI systems against hidden traps, ensuring that our digital helpers remain trustworthy even when the data they learn from isn't perfect.

Technical Summary: CLIP-Guided Backdoor Defense through Entropy-Based Poisoned Dataset Separation (CGD)

1. Problem Statement

Deep Neural Networks (DNNs) are vulnerable to backdoor attacks, where adversaries poison a small fraction of the training data to implant a trigger that forces the model to misclassify specific inputs into a target class. While existing defenses exist, they face significant limitations:

Clean Model-Based Defenses: Methods relying on self-supervised or semi-supervised learning (e.g., DBD, ASD) are computationally expensive and often ineffective against advanced clean-label backdoors, where labels remain correct but the trigger is embedded in the image.
Suspicious Model-Based Defenses: Methods analyzing the victim model's behavior (e.g., ABL, EP) struggle against clean-image backdoors, where poisoned images are visually indistinguishable from clean ones, making anomaly detection via data distribution or activations difficult.
Dependency on Clean Data: Many effective defenses (e.g., Fine-Pruning, MCR) assume the availability of a small, clean subset of data, which is often unavailable in real-world scenarios where training data is sourced externally without security guarantees.

2. Methodology: CLIP-Guided Backdoor Defense (CGD)

The authors propose CGD, a two-stage defense mechanism that leverages a publicly available, pre-trained vision-language model (CLIP) as a "weak but clean" classifier to separate poisoned data and guide model unlearning.

Core Insight

The method relies on an entropy-based analysis of predictions from two models:

CLIP (Weak but Clean): As a zero-shot classifier, CLIP is assumed to be free of the specific backdoor. It tends to produce high entropy (low confidence) on poisoned samples where the label does not match the visual content (label poisoning) or where the trigger disrupts semantic understanding.
Suspicious Model (Victim): The model trained on the poisoned dataset. In clean-label backdoors, the victim model learns a strong association between the trigger and the target label, resulting in low entropy (high confidence) predictions for poisoned samples, even if the trigger is subtle.

Stage 1: CLIP-Guided Meta Data Splitting

The framework generates an "entropy map" by evaluating each sample with both CLIP and the suspicious model.

Poison-Label Identification: Samples with high entropy from CLIP (indicating label-image mismatch) are flagged.
Clean-Label Identification: Samples with low entropy from the suspicious model (indicating over-confidence due to trigger association) are flagged.
Splitting Strategy: Using percentile ranks of entropy scores ( $\hat{S}_{CLIP}$ $\hat{S}_{C L I P}$ and $\hat{S}_{Model}$ $\hat{S}_{M o d e l}$ ), the dataset is partitioned into:
- Clean Subset ( $D_c$ ): Low CLIP entropy and high Model entropy.
- Triggered Subset ( $D_p$ ): High CLIP entropy OR low Model entropy.
- Mixed Subset: Remaining samples.

Stage 2: CLIP-Guided Backdoor Unlearning

The model is fine-tuned using a composite loss function on the separated subsets:

Relearning Loss ( $L_{re}$ ): Standard cross-entropy loss on $D_c$ to preserve clean accuracy.
Unlearning Loss ( $L_{un}$ ): Negative cross-entropy on $D_p$ to penalize confident predictions on triggered samples, disrupting the trigger-label association.
Distillation Loss ( $L_{distill}$ ): Kullback-Leibler divergence between the model's predictions and CLIP's predictions on $D_p$ . This guides the model to align with CLIP's "clean" zero-shot classifications, effectively neutralizing the backdoor influence.

Extension: Enabling Clean-Data-Based Defenses

A novel contribution of the paper is the ability to adapt defenses that traditionally require clean data (e.g., Fine-Pruning, ANP) to poisoned datasets. By using CGD to filter a high-quality "Split Clean" subset ( $D_c$ ) from the poisoned data, these methods can be applied without external clean data.

3. Key Contributions

Novel Defense Mechanism (CGD): Introduction of a defense that utilizes CLIP as a zero-shot classifier to identify poisoned samples via entropy analysis, addressing both label-poisoned and clean-label/clean-image backdoors.
Efficiency and Effectiveness: The method achieves high defense performance with minimal computational overhead (under 3 minutes post-training) and maintains high clean accuracy.
Bridging Defense Paradigms: Demonstrating that clean-data-based defenses can be effectively deployed on poisoned data by using CGD to extract a clean subset, removing the strict requirement for external clean data.
Robustness: The defense remains effective even when CLIP is weak (low zero-shot accuracy) or when CLIP itself is compromised by a backdoor.

4. Experimental Results

The authors evaluated CGD on four datasets (CIFAR-10, CIFAR-100, GTSRB, Tiny-ImageNet) against 11 state-of-the-art backdoor attacks.

Performance: CGD reduced Attack Success Rates (ASR) to below 1% (average 0.2% on CIFAR-10) across all attack categories, including clean-label and clean-image backdoors where other defenses failed.
Clean Accuracy (CA): The method maintained high clean accuracy with a maximum drop of only 0.3% on CIFAR-10.
Comparison: CGD outperformed existing poisoned-data-based defenses (e.g., ABL, DBD, EP) and, when used to filter data for clean-data-based defenses (e.g., ANP-SC, MCR-SC), achieved results comparable to or better than using 5% external clean data.
Efficiency: Total runtime for splitting and defense was approximately 1,154 seconds (including victim model training), significantly faster than self-supervised approaches like DBD (26,971 seconds).
Robustness Tests:
- Weak CLIP: Effective defense (ASR $\le$ 5%) even when CLIP's zero-shot accuracy was reduced to ~7.5%.
- Backdoored CLIP: CGD successfully removed backdoors from the victim model even when the guiding CLIP model itself contained a backdoor, without transferring the CLIP's backdoor.
- Adaptive Attacks: CGD resisted adaptive attacks (e.g., Rand-Patch, Feature Mixing Backdoor) that attempted to evade entropy detection.

5. Significance and Claims

The paper claims that CGD offers a practical, scalable, and highly efficient solution for real-world backdoor defense. Its significance lies in:

Overcoming Limitations: It successfully addresses the specific weaknesses of current defenses against clean-label and clean-image backdoors without the heavy computational cost of self-supervised learning.
Resource Efficiency: It eliminates the need for external clean data, a major bottleneck in practical deployment, by synthesizing a clean subset from the poisoned data itself.
Resilience: It provides strong security guarantees even under imperfect conditions (weak or compromised CLIP models), making it a robust choice for securing Deep Neural Networks in untrusted data environments.

The authors conclude that CGD represents a significant step forward in making deep learning applications more trustworthy by providing a defense that is both effective against advanced attacks and feasible for real-world deployment.