Diffusion or Non-Diffusion Adversarial Defenses: Rethinking the Relation between Classifier and Adversarial Purifier

This paper challenges the prevailing reliance on diffusion models for adversarial defense by demonstrating that non-diffusion purifiers can achieve superior robustness, transferability, and cross-dataset generalization, notably outperforming ImageNet-trained diffusion models when applied to ImageNet despite being trained only on CIFAR-10.

The Gist

Imagine you have a very smart security guard (the Classifier) whose job is to identify people entering a building. Sometimes, bad actors (the Adversarial Attacks) try to trick this guard by putting on tiny, almost invisible masks or wearing slightly weird clothes that make the guard think a friend is a stranger.

To stop this, researchers have been building "De-Mask Stations" (the Purifiers) to clean up the person's appearance before they reach the guard.

For a long time, the most popular De-Mask Station was built using Diffusion Models. Think of a Diffusion Model like a master painter who has only ever seen photos of cats. If you give this painter a picture of a dog, or a cat with a different fur color than the ones in their training book, the painter gets confused. They try to "fix" the image by painting it to look exactly like the cats in their book.

The Problem:
The paper argues that this "Master Painter" approach has a hidden flaw.

1. The Over-Correction: If the guard is used to seeing cats of all colors, but the painter forces every cat to look like the specific orange tabby from the training book, the guard might get confused. The painter changes the image so much that it no longer looks like the original person, even if the "bad mask" is gone.
2. The Color Issue: The paper found that these Diffusion painters are terrible at handling color changes. If you show them a red apple, they might try to turn it into a green apple because that's what they learned. This makes the security guard fail to recognize the apple.
3. The "One-Size-Fits-All" Failure: If you train this painter on small, blurry photos (like CIFAR-10) and then ask them to clean up a giant, high-definition photo (like ImageNet), they struggle. They can't generalize well to new, slightly different situations.

The Solution: The "Smart Editor" (MAEP)
The authors propose a new kind of De-Mask Station called MAEP (Masked AutoEncoder Purifier).

Instead of a painter who tries to recreate the whole image from scratch, imagine a Smart Editor who works like this:

• The Masking Game: The editor covers up random parts of the image (like putting sticky notes over parts of a face).
• The Guessing Game: The editor has to guess what's under the sticky notes based on the rest of the face.
• The Lesson: By doing this, the editor learns the structure and essence of the object (the "cat-ness" or the "apple-ness") rather than just memorizing specific colors or textures.

Why is the Smart Editor better?

1. It Respects the Original: When the editor cleans up the "bad mask," it only removes the noise. It doesn't try to repaint the whole picture. If the apple is red, it stays red.
2. It's a Chameleon: Because it learned the structure of things rather than just memorizing specific examples, it works great even if you show it a red apple when it was trained on green ones.
3. The Magic Result: The paper shows a stunning feat: They trained their "Smart Editor" on small, simple pictures (CIFAR-10), and then used it to clean up huge, complex photos (ImageNet) that it had never seen before. It actually performed better than the "Master Painters" that were specifically trained on those huge photos!

In Summary:
The paper says, "Stop trying to force every image to look like the training data (Diffusion). Instead, teach the system to understand the essence of the image so it can clean up noise without changing the identity of the object."

They proved that a simpler, non-diffusion method (MAEP) is more flexible, handles color changes better, and is actually more robust against tricky attacks than the fancy, popular diffusion models.

Technical Summary

1. Problem Statement

While diffusion models have recently emerged as powerful tools for adversarial purification (removing perturbations from input images before classification), this paper identifies a critical, often overlooked limitation: Classifier Generalization Loss.

• The Conflict: Diffusion-based purifiers are trained to map noisy images back to the training data distribution. However, classifiers are typically trained with data augmentation (e.g., color jitter, rotation) to learn robust features and generalize to unseen variations.
• The Issue: Diffusion models tend to "denoise" images toward the specific training distribution, effectively stripping away the natural variations (like color shifts) that the classifier expects. This creates a discrepancy between the purifier's output and the classifier's learned feature space.
• Consequence: When a diffusion purifier processes an image with slight variations (e.g., color changes) or when applied to a new dataset, it degrades the classifier's accuracy, even if the adversarial perturbation is removed. This limits defense transferability and robustness in real-world scenarios.

2. Methodology

The authors propose a shift from diffusion-based purification to a non-diffusion approach based on Masked Autoencoders (MAE) and Purification Loss.

A. Theoretical Analysis

• Distribution Shift: The authors argue that diffusion purifiers minimize the KL divergence between the input and the training distribution ($q(x)$). If the input $x'$ (e.g., a color-shifted image) deviates from $q(x)$, the purifier forces it back to $q(x)$, potentially altering semantic features the classifier relies on.
• Purification Loss vs. Clean Accuracy: They analyze the relationship between Purification Loss (reconstructing the clean image $x$ from adversarial $x_a$) and Clean Accuracy (ensuring the purifier doesn't harm clean images). They demonstrate that optimizing for purification loss ($\ell_1$ norm between $x$ and $P(x_a)$) implicitly preserves the direction of the adversarial perturbation ($-\delta_a$), allowing the model to remove attacks without significantly degrading clean image semantics.

B. Proposed Solution: Masked AutoEncoder Purifier (MAEP)

The authors introduce MAEP, a non-diffusion purifier that combines:

1. Masked Language Modeling (MLM) Objective: Inspired by MAE, the model masks parts of the input image and reconstructs them. This forces the model to learn robust patch representations and identify adversarial perturbations.
2. Purification Loss: A specific loss term ($\mathcal{L}_{purify}$) that reconstructs the clean image from the adversarial input, focusing on unmasked regions to ensure the removal of perturbations.
3. Reconstruction Loss: A standard reconstruction loss for masked regions to maintain semantic integrity.

The MAEP Loss Function:
$$\mathcal{L}_{MAEP} = \underbrace{\|M \odot x - M \odot g \circ f(M \odot x_a)\|}_{\text{Purification Loss (Unmasked)}} + \underbrace{\|(1-M) \odot x - (1-M) \odot g \circ f(M \odot x_a)\|}_{\text{Reconstruction Loss (Masked)}}$$

• $M$: Binary mask.
• $f, g$: MAE encoder and decoder.
• $x$: Clean image, $x_a$: Adversarial image.

3. Key Contributions

1. Identification of Generalization Loss: The paper is the first to systematically investigate and demonstrate that diffusion-based purifiers cause a loss in classifier generalization, particularly regarding color variations and dataset transferability.
2. Theoretical Explanation: They explain why diffusion models fail in these scenarios: the mismatch between the diffusion model's goal (generating natural images from a fixed distribution) and the classifier's goal (generalizing via augmentation).
3. MAEP Proposal: Introduction of a non-diffusion purifier (MAEP) that achieves state-of-the-art robustness while preserving clean accuracy and semantic details better than diffusion methods.
4. ColoredImageNet: The creation of a new benchmark dataset, ColoredImageNet, generated by applying color transfer techniques to ImageNet, specifically designed to evaluate the sensitivity of defenses to color shifts.

4. Experimental Results

The authors evaluated MAEP against state-of-the-art diffusion-based methods (DiffPure, ScoreOpt, MimicDiffusion) and other non-diffusion methods (DISCO, Anti-Adv) on CIFAR-10, CIFAR-100, and ImageNet.

• Standard Robustness (CIFAR-10/100):
  • MAEP achieves competitive or superior Robust Accuracy compared to DiffPure and ScoreOpt.
  • Crucially, MAEP maintains significantly higher Clean Accuracy than diffusion-based methods, which often suffer from accuracy drops due to over-smoothing or semantic loss.
• Sensitivity to Color Variations:
  • On ColoredImageNet, diffusion-based purifiers (DiffPure, ScoreOpt) showed an accuracy drop approximately twice as large as MAEP. This confirms the hypothesis that diffusion models are overly sensitive to color shifts.
• Defense Transferability (Cross-Dataset):
  • CIFAR-100 $\to$ CIFAR-10: DiffPure's robust accuracy dropped from ~89% to ~69% when transferred. MAEP maintained much higher performance.
  • CIFAR-10 $\to$ ImageNet (Zero-Shot): A MAEP model trained only on CIFAR-10 achieved ~75% clean accuracy on ImageNet, outperforming DiffPure and ScoreOpt (which were trained directly on ImageNet) by a significant margin.
• Image Quality:
  • MAEP preserves image details (high PSNR/SSIM) better than diffusion purifiers, which tend to introduce artifacts or alter textures significantly during the denoising process.

5. Significance and Conclusion

This paper challenges the prevailing trend of relying solely on diffusion models for adversarial defense. It highlights that more complex generative models do not necessarily equate to better defense, especially when the defense mechanism conflicts with the classifier's training objectives.

• Paradigm Shift: The work suggests that simpler, non-diffusion approaches based on reconstruction and purification losses can outperform complex diffusion models in terms of transferability and generalization.
• Practical Impact: MAEP offers a more practical solution for real-world deployment where models must handle unseen data distributions, color variations, and cross-dataset scenarios without requiring retraining on every specific dataset.
• Future Direction: The findings encourage the community to rethink the relationship between the purifier and the classifier, prioritizing the preservation of classifier generalization capabilities over pure generative fidelity.