Language Guided Adversarial Purification

This paper introduces Language Guided Adversarial Purification (LGAP), a novel defense framework that leverages pre-trained diffusion models and caption generators to guide the purification of adversarial images, achieving superior robustness against strong attacks without requiring specialized network training.

The Gist

Imagine you have a very smart, but slightly gullible, security guard (the AI Classifier) whose job is to identify objects in photos. Usually, he's great at spotting a "Panda" or an "Elephant."

However, bad actors (adversaries) have learned a trick: they can add invisible "static" or "noise" to a photo. To the human eye, the photo looks fine, but to the security guard, that tiny bit of noise makes him scream, "That's not a panda! That's a ship!" and he gets fooled.

For a long time, the solution was to train the guard by showing him thousands of these tricked photos. But this is like trying to teach a guard every single possible disguise in the world; it takes forever, costs a fortune, and if the bad guys invent a new trick, the guard is caught off guard again.

Enter LGAP: The "Translator and Artist" Team

The paper introduces a new defense called LGAP (Language Guided Adversarial Purification). Instead of training the guard harder, they put a Translator and a Master Artist in front of him.

Here is how the process works, using a simple analogy:

1. The Translator (BLIP)

First, the suspicious photo (the one with the invisible noise) is handed to a Translator. This is a pre-trained AI that is really good at looking at a picture and writing a sentence about it.

• The Magic: Even if the photo has been tampered with to trick the security guard, the Translator is so smart it ignores the tiny noise. It looks at the photo and says, "I see a fire truck."
• Why this matters: The bad guys tried to make the guard think it's a ship, but the Translator correctly identifies it as a truck.

2. The Master Artist (Diffusion Model)

Next, the Translator's sentence ("A fire truck") is passed to a Master Artist. This artist uses a special technique called a "Diffusion Model."

• The Process: Imagine the artist takes the messy, noisy photo and starts "painting over" it. But they don't just paint randomly. They use the Translator's sentence as a strict guide.
• The Result: The artist says, "Okay, the caption says 'Fire Truck,' so I will reconstruct the image to look exactly like a clean, perfect fire truck, removing all the invisible noise the bad guys added."
• The output is a brand new, clean image that looks just like the original object, but without the trickery.

3. The Security Guard

Finally, this fresh, clean image is handed to the security guard. Since the noise is gone and the image is now a perfect "Fire Truck," the guard correctly identifies it.

Why is this a big deal?

Most previous methods were like trying to memorize every possible disguise. They required massive amounts of computing power and specific training for every new type of attack.

LGAP is different because:

• It uses what it already knows: It uses models (the Translator and the Artist) that were already trained on huge amounts of data (like the entire internet). We don't need to retrain them from scratch.
• It's a generalist: Because the Translator understands language and the Artist understands the world, they can handle any type of noise, even ones the system has never seen before.
• It's efficient: It's like hiring a team of experts who already know their jobs, rather than spending years training a rookie from scratch.

The Bottom Line

The researchers tested this on famous image datasets (like CIFAR and ImageNet). They found that LGAP was incredibly good at cleaning up "poisoned" images and stopping them from fooling the AI.

In short, LGAP doesn't fight the noise directly; it uses a "description" of the image to rebuild the picture from scratch, effectively washing away the attack. It's a smarter, faster, and more flexible way to keep AI safe.

Technical Summary

1. Problem Statement

Deep neural networks, particularly in computer vision, are highly vulnerable to adversarial perturbations—subtle, often imperceptible changes to input images that cause models to misclassify data. Existing defense strategies face significant limitations:

• Adversarial Training: Requires extensive training on specific adversarial examples. It is computationally expensive and often fails to generalize against novel attack strategies not seen during training.
• Adversarial Purification (Generative Models): While methods using GANs, Energy-Based Models (EBMs), and Diffusion models show promise, they are often computationally intensive. Many require training the purification model specifically on the target dataset or extensive fine-tuning, limiting their scalability and generalizability.
• Modality Limitation: Most purification techniques rely solely on image data, ignoring the potential of multi-modal information (e.g., text) to guide the restoration of clean semantic content.

2. Methodology: Language Guided Adversarial Purification (LGAP)

The authors propose LGAP, a novel framework that leverages pre-trained Vision-Language models and Diffusion models to purify adversarial inputs without requiring specialized training of the purification network. The process consists of three main stages:

A. Image Captioning (Semantic Extraction)

• Tool: A pre-trained BLIP (Bootstrapping Language-Image Pre-training) model is used.
• Process: Given an input image (which may be adversarial), BLIP generates a text caption describing the visual content.
• Key Insight: Even when an image is adversarially perturbed (causing a classifier to misclassify it, e.g., "truck" $\to$ "ship"), the BLIP model often correctly identifies the true semantic content (e.g., still generating "truck"). This provides a robust "ground truth" signal derived from large-scale pre-training.

B. Diffusion Purification (Guided Generation)

• Tool: A pre-trained Latent Diffusion Model (LDM).
• Process:
  1. The adversarial image is encoded into a latent vector ($z_0$).
  2. A forward diffusion process adds noise to create a noisy latent vector ($z_t$).
  3. Reverse Process: The model attempts to recover the clean latent vector ($z_0$) from the noisy state. Crucially, this reverse process is conditioned on the text caption generated by BLIP.
  4. The diffusion model uses cross-attention layers to integrate the text embedding ($C$) with the image generation process: $z_t = g_\theta(z_{t+1}, t, \epsilon_t, C)$.
• Mechanism: The text caption acts as a strong semantic guide, steering the diffusion model to reconstruct an image that matches the true semantic description (e.g., "a truck") rather than the corrupted adversarial features.

C. Classification and Fine-tuning

• The reconstructed (purified) image is passed to the target classifier.
• Training Efficiency: Unlike traditional adversarial training, LGAP requires minimal training. The diffusion and captioning models are frozen (pre-trained). Only the target classifier undergoes a brief fine-tuning (e.g., 15 epochs) on the purified clean samples to adapt to the new input distribution.

3. Key Contributions

1. Novel Framework: Introduction of LGAP, the first framework to utilize language guidance (via captions) to direct the adversarial purification process in diffusion models.
2. Zero-Shot/Pre-trained Efficiency: The method leverages the generalizability of models trained on massive datasets (BLIP and LDM). It does not require training the purification model on specific adversarial examples or the target dataset, significantly reducing computational overhead.
3. Robustness to Novel Attacks: By relying on semantic understanding rather than specific attack vectors, the method demonstrates strong defense against both standard and adaptive attacks (including BPDA and EOT).
4. Scalability: The approach is classifier and attack-agnostic, making it versatile for different architectures and threat models.

4. Experimental Results

The authors evaluated LGAP on CIFAR-10, CIFAR-100, and ImageNet against strong $L_\infty$ norm attacks (PGD) and adaptive attacks (BPDA, EOT).

• CIFAR-10:
  • Achieved 71.68% robust accuracy under preprocessor-blind PGD attacks.
  • Outperformed 7 out of 10 existing methods (including adversarial training and other purification techniques) while maintaining high natural accuracy (90.03%).
  • Significantly outperformed methods that required extensive training of purification networks (e.g., Yoon et al., Hill et al.).
• CIFAR-100:
  • Achieved 39.82% robust accuracy, outperforming several adversarial training baselines (e.g., Madry et al. at 25.47%) with lower computational cost.
• ImageNet:
  • Demonstrated efficacy on large-scale data, achieving 45.31% robust accuracy against strong adaptive BPDA-40 attacks.
  • The performance gain is attributed to the diffusion model's pre-training on ImageNet, highlighting the benefit of large-scale data.

5. Significance and Conclusion

The paper establishes a new direction in adversarial defense by shifting from attack-specific training to semantic-guided purification.

• Efficiency: It drastically reduces the computational burden associated with adversarial defense by eliminating the need to train score networks or diffusion models from scratch.
• Generalizability: It proves that models trained on large, diverse datasets (like BLIP and LDM) possess inherent robustness that can be harnessed for defense without specialized fine-tuning of the generative components.
• Future Impact: LGAP underscores the potential of multi-modal learning (Vision + Language) to solve security vulnerabilities in AI, suggesting that leveraging semantic understanding is a powerful strategy for enhancing model robustness.