🤖 machine learning

Improving Certified Robustness via Adversarial Distillation

The paper introduces AD-CERT, a certified training framework that combines adversarial distillation from an empirically robust teacher with Interval Bound Propagation to achieve state-of-the-art certified robustness while significantly improving the trade-off between standard accuracy and formal verification.

Original authors: Matteo Melis, Jesus Martinez Del Rincon, Vishal Sharma

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Matteo Melis, Jesus Martinez Del Rincon, Vishal Sharma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart student (a computer program) who needs to learn how to recognize pictures, like cats or dogs. But there's a catch: someone is trying to trick the student by adding tiny, almost invisible "noise" to the pictures—like a speck of dust that makes a cat look like a dog to the computer. This is called an adversarial attack.

The paper introduces a new way to train this student, called AD-CERT, to make them both smart at recognizing normal pictures and unshakeable against these tricks. Here is how it works, broken down into simple concepts:

1. The Two Problems: Being Strong vs. Being Safe

In the world of AI training, there are usually two different approaches, and they often fight each other:

The "Street Fighter" (Adversarial Training): This method trains the student by showing them thousands of tricked pictures. The student learns to fight back and get the right answer even when the picture is messed up. This makes the student very strong in real-world tests, but their internal logic becomes so messy and complex that no one can prove why they are safe. It's like a fighter who wins every match but has no rulebook to explain their moves.
The "Safety Inspector" (Certified Training): This method tries to mathematically prove that the student will never be tricked, no matter what. It uses a strict "safety net" (called IBP) that checks every possible way a picture could be changed. This creates a model that is provably safe, but often, the student becomes too cautious and starts making mistakes on normal, clean pictures. It's like a safety inspector who is so worried about accidents that they refuse to let anyone drive at all.

2. The New Solution: The "Tutor and the Safety Net"

The authors of this paper realized they could have the best of both worlds by using Knowledge Distillation. Think of this as a master teacher and a student.

The Teacher: They first train a "Teacher" model using the "Street Fighter" method. This teacher is incredibly good at handling tricked pictures (empirically robust).
The Student: They then train a "Student" model. But instead of just fighting the tricks alone, the student listens to the Teacher.

The AD-CERT Recipe:
The student learns using a two-part recipe:

The Teacher's Whisper (Adversarial Distillation): The student looks at a tricked picture and tries to match the thought process (the "logits" or internal scores) of the Teacher. The Teacher says, "Even though this picture is messed up, I'm 90% sure it's a cat." The student learns to think the same way. This gives the student the Teacher's street-smart strength.
The Safety Net (IBP): At the same time, the student is forced to run through the strict "Safety Inspector" math. This ensures that the student's final answer is mathematically guaranteed to be safe against any possible trick.

3. Why This is Special

Usually, mixing these two methods is hard because they speak different languages. The "Street Fighter" uses hard, specific examples, while the "Safety Inspector" uses broad, fuzzy math.

The paper's big insight is that the Teacher's "whisper" acts as a surrogate (a stand-in) for the hard math.

Imagine the Teacher is a seasoned detective who knows exactly how a criminal might hide.
The Student doesn't need to guess how the criminal hides; they just copy the Detective's intuition.
Meanwhile, the Safety Net (IBP) ensures that even if the Detective is wrong, the math proves the Student is safe.

4. The Results

The paper tested this on standard image datasets (like MNIST, CIFAR-10, and TinyImageNet).

The Outcome: The AD-CERT student became the champion. It achieved the highest "certified accuracy" (the ability to be mathematically proven safe) compared to all other previous methods.
The Trade-off: It didn't lose much on "standard accuracy" (how well it sees normal pictures). In fact, it was often better than other safe methods.
The Catch: There is still a gap between the "Street Fighter" Teacher and the "Safe" Student. The Student is very safe, but sometimes it's not quite as good at recognizing normal pictures as the Teacher was. The authors admit that closing this gap on very difficult puzzles is the next big challenge.

Summary Analogy

Think of training a security guard for a bank.

Old Method A: You train the guard by throwing rocks at them and teaching them to dodge. They get good at dodging, but you can't prove they won't miss a subtle trick.
Old Method B: You teach the guard a strict rulebook that covers every possible angle. They are perfectly safe, but they are so rigid they trip over their own feet on normal days.
AD-CERT: You hire a legendary, battle-hardened guard (the Teacher) to teach a new recruit (the Student). The new recruit learns the legendary guard's instincts for dodging rocks (Distillation) while simultaneously being tested against a strict, mathematical safety checklist (IBP). The result is a guard who is both instinctively tough and mathematically proven to be safe.

Technical Summary: Improving Certified Robustness via Adversarial Distillation

Problem Statement
Neural networks are vulnerable to adversarial examples, where small input perturbations cause misclassification. While adversarial training (AT) improves empirical robustness, the resulting models are difficult to formally verify. Conversely, certified training methods, which optimize upper bounds on the worst-case loss (typically via Interval Bound Propagation, or IBP), produce verifiable models but often suffer from degraded standard accuracy. Recent state-of-the-art approaches attempt to balance this by interpolating between adversarial lower bounds and IBP upper bounds. However, the trade-off between standard accuracy and certified accuracy remains a significant challenge, particularly as perturbation radii increase.

Methodology: AD-CERT
The authors propose AD-CERT (Adversarial Distillation for Certification), a novel training objective that combines adversarial knowledge distillation with IBP bounds. The core hypothesis is that transferring adversarial knowledge from an empirically robust teacher to a certified student can serve as an effective surrogate for the adversarial lower bound, improving the standard-certified accuracy trade-off.

The AD-CERT loss function is formulated as a linear combination of two terms:
$L_{AD-CERT} := (1 - \alpha) \underbrace{KL(P_{\theta_t}(x) \parallel P_{\theta_s}(x_{adv}))}_{\text{Adversarial Distillation}} + \alpha \underbrace{L_{CE}(-z^{\Delta}_{\theta_s}(x, y), y)}_{\text{IBP}}$

Where:

$f_{\theta_t}$ is a frozen teacher network trained via standard adversarial training (PGD).
$f_{\theta_s}$ is the student network being trained for certification.
$x_{adv}$ is an adversarial example generated against the student.
The Adversarial Distillation (AD) term minimizes the Kullback-Leibler (KL) divergence between the teacher's predictive distribution on clean inputs ( $P_{\theta_t}(x)$ ) and the student's predictive distribution on adversarial inputs ( $P_{\theta_s}(x_{adv})$ ).
The IBP term computes the standard certified upper bound loss using Interval Bound Propagation.
$\alpha \in [0, 1]$ balances the empirical and certified objectives.

Theoretical Insights
The paper provides a theoretical analysis characterizing AD-CERT:

Surrogate Lower Bound: The AD term is shown to be equivalent to adversarial training with soft teacher labels, acting as a "teacher-guided lower bound surrogate." Unlike standard AT which optimizes a hard lower bound, AD-CERT uses the teacher's soft labels to guide the student's behavior on adversarial examples.
Scalarisation: The objective is a scalarisation between a soft-label adversarial lower bound surrogate and a certified IBP upper bound.
Geometric Difference: Unlike standard AT, which pushes for arbitrarily large class margins, the AD branch induces a finite target in the logit space defined by the teacher's predictive distribution ( $\Delta z^*_{ij} = \log(P_{\theta_t}(x)_i / P_{\theta_t}(x)_j)$ ). This suggests a smoother optimization landscape.

Experimental Results
The authors evaluated AD-CERT on MNIST, CIFAR-10, and TinyImageNet using the CTBench library and $\alpha,\beta$ -CROWN verifier.

State-of-the-Art Performance: AD-CERT achieves state-of-the-art certified accuracy across all evaluated ReLU-based architectures.
- On MNIST, it provides consistent, modest improvements over existing methods.
- On CIFAR-10, it improves the best prior ReLU-based certified accuracy by an average of 0.37 percentage points, with the largest gain at $\epsilon = 2/255$ .
- On TinyImageNet, AD-CERT achieves the highest certified accuracy among all compared methods, substantially improving standard accuracy over previous non-distillation certified training methods.
Comparison with Distillation Methods: In a controlled comparison against CC-DIST (which performs robust feature-space distillation), AD-CERT (logit-level distillation) achieves higher certified accuracy, while CC-DIST retains slightly higher standard accuracy. AD-CERT improves certified accuracy over CC-DIST by up to 5.40 percentage points in specific settings.
Ablation Studies:
- Removing the IBP component to test pure distillation techniques reveals that matching a teacher only on clean inputs (Natural KL) fails to transfer adversarial robustness.
- Adversarial logit-level distillation (AD) preserves teacher-level robustness more effectively than feature-space distillation or other adversarial distillation variants (like RSLAD) when used as the empirical endpoint.
- Sensitivity analysis of the IBP coefficient $\alpha$ confirms that reducing IBP strength improves standard accuracy but eventually degrades certified accuracy, consistent with trends in expressive loss literature.

Significance and Claims
The paper claims that AD-CERT successfully bridges the gap between empirical robustness and formal verification. By decoupling the empirical endpoint (using a teacher-guided logit-level distillation) from the certified endpoint (using pure IBP), the method avoids the complexities of tightly coupled expressive losses found in prior work.

The authors emphasize that while AD-CERT achieves superior certified performance, a significant gap remains between the standard/adversarial accuracy of the empirically robust teacher and the certifiably robust student, particularly on larger perturbation radii (e.g., CIFAR-10 at $\epsilon = 8/255$ ). The paper concludes that while logit-level distillation is an effective mechanism for transferring robustness, future work is needed to develop stronger methods for transferring empirical robustness into certifiable robustness without sacrificing standard accuracy.