Technical Summary: An Empirical Study of the Influence of Adversarial Fine-Tuning on Compressed Neural Networks

Problem Statement

Deep learning (DL) models are increasingly integrated into safety-critical applications, necessitating robustness against adversarial attacks. However, ensuring this robustness typically requires adversarial training, which introduces significant computational overhead due to the generation of adversarial examples during training. Conversely, model compression techniques (such as pruning and quantization) are employed to improve computational efficiency and reduce energy consumption but often degrade model performance and, crucially, adversarial robustness. This creates a conflict between the objectives of computational efficiency and adversarial robustness. The central question addressed by this work is whether adversarial robustness can be efficiently achieved for already compressed neural networks without the full cost of adversarial training from scratch.

Methodology

The authors propose a two-stage approach: first compressing a standard model, and then applying adversarial fine-tuning rather than full adversarial training. The study focuses on two primary compression methods:

1. Structured Pruning: Specifically, $\ell_1$-norm based filter pruning for Convolutional Neural Networks (CNNs). Filters with the lowest $\ell_1$-norm are removed, along with corresponding output features in subsequent layers.
2. Quantization: Post-Training Quantization (PTQ) is utilized, employing symmetric quantization for weights and asymmetric quantization for activation maps to balance precision and computational simplicity.

The experimental framework involves three main phases:

• Baseline Training: Training standard models ($f_{st}$) and fully adversarially trained models ($f_{rb}$) from scratch.
• Compression: Applying pruning or quantization to the trained models.
• Fine-Tuning: Applying either standard fine-tuning ($T_{st}$) or adversarial fine-tuning ($T_{ad}$) to the compressed models. Adversarial fine-tuning involves minimizing the adversarial loss (using Projected Gradient Descent, PGD) for a limited number of epochs (typically three) starting from the pre-trained, compressed weights.

The study evaluates these methods across multiple datasets (MNIST, FashionMNIST, SVHN, CIFAR-10, CIFAR-100, TinyImageNet) and architectures (WideResNet, Vision Transformer, and custom CNNs). Robustness is evaluated using AutoAttack (specifically APGD-CE and APGD-DLR) to provide a reliable lower bound on robust accuracy.

Key Contributions

1. Adversarial Fine-Tuning for Compressed Models: The paper introduces and validates adversarial fine-tuning as an efficient mechanism to restore and enhance the robustness of compressed models.
2. Comprehensive Empirical Study: The authors conduct extensive experiments comparing standard fine-tuning versus adversarial fine-tuning on models subjected to both pruning (at various sparsity ratios) and quantization (at various bit-widths).
3. Fair Compression Comparison: The study systematically tunes compression levels (e.g., matching 80% sparsity pruning with INT8 quantization) to ensure fair comparisons between different compression techniques based on test performance.

Results

The experimental results demonstrate that adversarial fine-tuning of compressed models yields significant improvements in robustness while maintaining computational efficiency:

• Robustness Recovery: Compressed standard models ($f^c_{st}$) that initially exhibit near-zero robustness (e.g., ~0% on CIFAR-10) can achieve robustness comparable to fully adversarially trained models ($f^c_{rb}$) after only three epochs of adversarial fine-tuning. For instance, on MNIST with WideResNet, adversarial fine-tuning of a pruned standard model increased robustness from ~22% (with standard fine-tuning) to ~92%, matching the performance of a model adversarially trained from scratch.
• Efficiency Gains: The computational cost of adversarial fine-tuning is substantially lower than full adversarial training. On CIFAR-10, the time required to achieve robustness dropped from approximately 118 minutes (full adversarial training) to roughly 14 minutes for compressed models, and further down to ~10 minutes for adversarial fine-tuning of compressed models.
• Generalizability: The positive effects of adversarial fine-tuning were observed across different datasets, model architectures (CNNs, ViTs), and compression methods (pruning, quantization, and tensor factorization).
• Standard vs. Robust Models: The study notes that while standard fine-tuning helps recover test accuracy for pruned models, it often fails to recover robustness. Conversely, adversarial fine-tuning recovers both test accuracy and robustness, often bringing the performance of compressed standard models close to that of compressed robust models.

Significance and Claims

The paper claims that adversarial fine-tuning serves as a "lightweight substitute" for full adversarial training, particularly when combined with model compression. The authors argue that this approach allows for the compounding of efficiency gains: the model is first made efficient through compression, and then made robust through a computationally cheap fine-tuning process.

The significance lies in resolving the trade-off between efficiency and safety. The results suggest that it is possible to deploy models that are both computationally efficient (due to compression) and robust against adversarial attacks (due to fine-tuning) without incurring the prohibitive costs of training robust models from scratch. The authors conclude that this method provides a practical pathway for deploying safe, resource-constrained deep learning models in real-world applications.

Note: The paper acknowledges limitations, such as the dependency of the optimal number of fine-tuning epochs on specific tasks and datasets, and notes that for highly complex datasets like CIFAR-100 and TinyImageNet, the robustness gains were less pronounced compared to simpler datasets.