🤖 machine learning

Improving the Robustness/Accuracy Tradeoff Against Adversarial Attacks Using Information Bottleneck Distillation Through Dual Teachers

This paper proposes an enhanced Information Bottleneck Distillation framework that integrates a clean teacher model alongside a robust teacher via cross-layer attention to improve clean-input accuracy while maintaining adversarial robustness, thereby achieving a superior tradeoff compared to existing methods.

Original authors: Vincent Ryusuke Takahashi, Yoshinari Takeishi, Jun'ichi Takeuchi, Kave Salamatian

Published 2026-07-31

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Vincent Ryusuke Takahashi, Yoshinari Takeishi, Jun'ichi Takeuchi, Kave Salamatian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a brilliant but fragile student to recognize animals in a photo. You show them a picture of a cat, and they say, "Cat!" Perfect. But then, a mischievous trickster sneaks in and adds a few invisible pixels to the photo—so subtle that your eyes can't see the change, but the computer's brain goes haywire and screams, "Toaster!" This is the world of adversarial attacks, a major headache for the artificial intelligence systems running our self-driving cars and security cameras. These systems are incredibly smart at spotting patterns, but they are surprisingly easily fooled by tiny, crafted tricks.

To fix this, scientists have tried a technique called adversarial training, which is like showing the student thousands of these tricked photos so they learn to ignore the noise. However, there's a catch: the more you train the student to be tough against tricks, the worse they get at recognizing normal, everyday photos. It's a classic trade-off: make the student a bodyguard, and they might forget how to be a librarian. Another idea, called knowledge distillation, involves having a super-smart "teacher" AI teach a smaller "student" AI. The teacher knows all the answers, and the student tries to copy them. Recently, a method called Information Bottleneck Distillation (IBD) tried to combine these ideas, using a tough teacher to make the student robust. But even this method struggled to keep the student's "clean" accuracy high while making them tough.

This paper introduces a clever new twist to that story. The authors, Ryusuke Takahashi and his team, realized that maybe one teacher isn't enough. They propose a dual-teacher system. Imagine the student now has two mentors: a "Robust Teacher" who is an expert at spotting tricked photos, and a "Clean Teacher" who is an expert at spotting normal, perfect photos. Instead of just copying one, the student learns from both simultaneously. The researchers found that by letting the student listen to the Clean Teacher for normal pictures and the Robust Teacher for tricked ones, they could create a student that is tough against attacks without forgetting how to recognize normal images.

The team tested this idea on two famous image datasets, CIFAR-10 and CIFAR-100, which are like giant photo albums of everyday objects. They compared their new "Double Distillation" and "Joint Distillation" methods against the old single-teacher approach. The results were promising: their new method improved the accuracy on normal, clean photos significantly while keeping the defense against attacks just as strong as before. In fact, when they measured the "harmonic mean"—a score that rewards a model for being good at both tasks rather than just one—their new methods beat the old standard. They also discovered that this trick works best when the student model is large enough to handle information from two different teachers; if the student is too small, it gets overwhelmed. While the attention mechanism (the part of the system that decides which teacher to listen to) sometimes became less active during training, the authors suggest that even a short burst of this dual-teacher guidance was enough to make the student smarter. Ultimately, the paper suggests that having a balanced team of teachers helps AI find a better balance between being smart and being safe.

Technical Summary: Improving the Robustness/Accuracy Tradeoff Against Adversarial Attacks Using Information Bottleneck Distillation Through Dual Teachers

Problem Statement
Deep neural networks (DNNs) are highly effective for classical machine learning tasks but remain vulnerable to adversarial attacks, where imperceptibly perturbed inputs cause incorrect predictions. While adversarial training enhances robustness, it often degrades classification accuracy on clean (non-adversarial) inputs, creating a significant trade-off. Existing countermeasures, specifically Information Bottleneck Distillation (IBD) introduced by Kuang et al., improve robustness but similarly suffer from reduced clean accuracy. The core problem addressed in this work is how to improve the balance between clean accuracy and adversarial robustness without sacrificing one for the other.

Methodology
The authors propose an extension to the IBD framework by introducing a dual-teacher distillation architecture. Unlike the original IBD, which utilizes a single "robust teacher" trained via adversarial training, the proposed method incorporates a second "clean teacher" trained exclusively on clean inputs. The student model learns from both teachers through two distinct architectures:

Double Distillation (DD): The student model aligns its intermediate representations with the robust teacher's features (using adversarial inputs) while simultaneously minimizing cross-entropy loss against the clean teacher's soft labels (using clean inputs). Feature distillation is performed solely using the robust teacher.
Joint Distillation (JD): The student model distills knowledge from both teachers simultaneously. It aligns intermediate features with the robust teacher for adversarial inputs and with the clean teacher for clean inputs. Crucially, the attention weights linking teacher features to student features are learned separately for the clean and robust distillation terms.

Both methods employ a cross-layer attention matrix to weight the differences between teacher and student latent features, approximating the Information Bottleneck objective. The loss function combines cross-entropy terms for clean and adversarial inputs with a feature-matching term weighted by these learned attention matrices.

Key Contributions

Dual-Teacher Framework: The introduction of a "clean teacher" alongside the standard "robust teacher" to specifically target the preservation of clean sample classification performance while maintaining robustness.
Architectural Variants: The proposal and evaluation of Double Distillation and Joint Distillation strategies, demonstrating how different feature alignment strategies impact the accuracy-robustness trade-off.
Attention Mechanism Analysis: An empirical investigation into the behavior of the attention module under different regularization settings (global vs. selective). The authors observe that under global regularization, attention weights tend to become uniform, whereas selective regularization allows for more concentrated, layer-specific attention patterns.
Scalability Analysis: An examination of how student model size affects performance, noting that the proposed methods become increasingly effective as the student model capacity increases relative to the teacher.

Experimental Results
Experiments were conducted on CIFAR-10 and CIFAR-100 datasets using Pre-Activation ResNet-18 students and WideResNet teachers. Adversarial attacks included FGSM, PGD, CW, and Auto-Attack.

Performance Trade-off: Both DD and JD methods outperformed the baseline IBD in terms of clean accuracy while maintaining comparable robust accuracy.
Harmonic Mean: The proposed methods achieved higher harmonic means (the metric used to evaluate the balance between clean and robust accuracy) compared to the baseline IBD.
Comparison with SOTA: The methods were competitive with state-of-the-art approaches, including the recent dual-teacher framework B-MTARD. On CIFAR-10, the Joint Distillation (JD) method achieved the highest robust accuracy across all evaluated attacks and the highest harmonic mean under PGD, CW, and AA attacks. On CIFAR-100, JD consistently secured the highest harmonic mean, indicating an optimal trade-off.
Model Size Impact: The proposed methods showed particular efficacy with larger student models (size $\ge$ 1/4 of the teacher), where robust accuracy was comparable to or exceeded that of IBD. Conversely, for very small models, IBD performed slightly better.
Regularization Sensitivity: The study found that the "selective regularization" setup (removing L2 regularization from biases and distillation layers) favored the baseline IBD, while the "global regularization" setup allowed the proposed DD and JD methods to demonstrate their maximum efficacy.

Significance and Claims
The paper claims that the proposed dual-teacher distillation framework successfully mitigates the degradation of clean accuracy often associated with robust training. By leveraging a clean teacher to guide the student on non-adversarial data and a robust teacher for adversarial resilience, the method improves the overall utility of the model in real-world scenarios where both clean and adversarial inputs may occur. The authors assert that their approach offers a competitive alternative to existing state-of-the-art defenses, particularly in maximizing the harmonic mean between clean and robust accuracy, and that the benefits are most pronounced when the student model has sufficient capacity to process information from two distinct teacher sources.

Technical Summary: Improving the Robustness/Accuracy Tradeoff Against Adversarial Attacks Using Information Bottleneck Distillation Through Dual Teachers

More like this