GRAPE: Guided Parameter-Space Evolution for Compact Adversarial Robustness

Technical Summary: GRAPE (Guided Parameter-Space Evolution)

Problem Statement
Adversarial Training (AT) is the standard approach for enhancing neural network robustness, yet it typically operates under the assumption that the optimizable parameter space remains fixed throughout the training process. While larger model capacities often correlate with improved robust accuracy, prior research indicates that robustness does not increase monotonically with parameter count. Furthermore, existing methods often assume that the static total number of parameters defined before training solely determines the model's robust capability. This paper challenges the implicit assumption that the entire parameter space must be exposed from the beginning of training. It posits that the order in which the parameter space is exposed and expanded during training may significantly influence the final robust solution, even if the final network architecture remains unchanged.

Methodology: GRAPE
The authors propose GRAPE (Guided Parameter-Space Evolution), a training framework designed to achieve compact adversarial robustness by treating robust learning as a dynamic process of parameter-space exposure, stabilization, and expansion. GRAPE consists of two primary components:

1. Parameter-Space Stabilization: Borrowing from local smoothing techniques like AWP and ISEAT, this component stabilizes the local optimization process within the current parameter space. It employs an instance-adaptive smoothing term ($L_{stab}$) in the training objective to reduce sharp loss landscapes, ensuring stable robust optimization trajectories before and after capacity expansion.
2. Guided Progressive Hidden Expansion: Unlike fixed-structure AT, GRAPE progressively releases new optimizable dimensions during training.
   • Mechanism: The framework starts with a reduced hidden capacity (initially 0.25× the standard width) and expands specific residual blocks over time. New hidden channels are initialized to produce near-zero output (via zero-initialized output projections) to preserve the function of the model immediately after growth, ensuring a smooth transition.
   • Guidance Strategy: To determine where to expand capacity, GRAPE utilizes an Adversarial Spectral Utilization Score. This score is computed on the post-shortcut outputs of residual blocks using the effective rank (derived from the spectral entropy of singular values) of adversarial representations. The score ($u_l$) represents the adversarial effective dimension supported per unit of hidden capacity. Blocks with higher scores (indicating higher capacity pressure) are selected for expansion.

The training objective evolves from a smaller parameter space $\Theta_0$ to a larger one $\Theta_T$ ($\Theta_0 \subset \Theta_1 \subset \dots \subset \Theta_T$), guided by the spectral utilization score to allocate new capacity to high-pressure modules.

Key Contributions

• Reconceptualizing Robust Capacity: The paper argues that the robust capability of a model should not be explained solely by the static parameter count but must account for the progressive exposure and expansion path of the parameter space during training.
• The GRAPE Framework: A novel training method combining local parameter-space stabilization with guided progressive hidden expansion, allowing the model to absorb new degrees of freedom gradually rather than searching a full high-dimensional space from the start.
• Empirical Validation of Exposure Path: Through a "Stabilized Sequential Grow" variant (which preserves the final ResNet-18 architecture but grows blocks in a fixed order), the authors demonstrate that the gain in robustness is not solely due to architectural differences but is intrinsically linked to the parameter-space exposure path.
• Compactness and Efficiency: The method achieves higher robust accuracy with fewer parameters under a matched computation budget (FLOPs), suggesting that guided evolution can form more compact and robust parameter configurations.

Experimental Results
Experiments were conducted on the CIFAR-10 dataset under the standard $\ell_\infty$ threat model ($\epsilon = 8/255$) using ResNet-18 backbones.

• Performance: Compared to fixed-structure ResNet-18 AT (baseline PGD-20 accuracy: 51.70%), GRAPE achieved 56.94% PGD-20 robust accuracy. This represents a relative improvement of approximately 10.1%.
• Efficiency: This improvement was achieved with a nearly matched computation budget (FLOPs ratio of 1.009×) while reducing the parameter count by approximately 21.4% (Params Ratio: 0.786×).
• Ablation Insights:
  • Stabilized Seq-Grow: A variant that grows blocks sequentially (B1 to B8) without score guidance but preserves the final ResNet-18 architecture achieved 56.52% accuracy. This confirms that the progressive exposure path itself contributes to the gain, independent of the final architecture shape.
  • Score Guidance: GRAPE (score-guided) outperformed the sequential grow variant, indicating that allocating capacity to high-pressure modules based on spectral utilization further enhances efficiency and robustness.
  • Stabilization: The combination of stabilization and guided growth (GRAPE) was shown to be superior to guided growth alone (AT-Grow).

Significance and Claims
The paper claims that the robust capability of adversarially trained models is not merely a function of the final static architecture or total parameter count. Instead, it is significantly influenced by how the parameter space is organized, exposed, and expanded during the training process. By mimicking the high-level principle of developmental plasticity—where representational pathways are progressively formed and stabilized—GRAPE demonstrates that a more compact and robust parameter-space configuration can be formed under matched computation budgets. The work suggests that the "capacity release path" is a critical factor in adversarial optimization, offering a new perspective beyond simply enlarging model capacity or modifying the adversarial objective.

The authors note limitations, including the focus on CIFAR-10 and ResNet-18, the need for systematic hyperparameter studies, and the lack of deep theoretical analysis regarding how parameter-space evolution alters adversarial optimization geometry. They emphasize that the goal is not parameter reduction for its own sake, but the formation of a more efficient robust configuration through dynamic evolution.