Technical Summary: Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimization

1. Problem Statement

Deep Neural Networks (DNNs) often suffer from over-parameterization and non-convex loss landscapes, leading to numerous local optima with varying generalization capabilities. The prevailing hypothesis suggests that "flat minima" (regions with low curvature) generalize better than "sharp minima." Sharpness-Aware Minimization (SAM) was proposed to address this by minimizing the worst-case loss within a fixed parameter-space radius $\rho$.

However, the authors identify a fundamental mechanism-level mismatch in SAM and its variants:

• The Ideal: Flatness is inherently a second-order property, characterized by the local curvature (Hessian) of the loss landscape.
• The Reality: SAM constructs its adversarial objective using a first-order linear approximation. Within a fixed radius $\rho$, the perturbation is aligned with the gradient direction. Consequently, the resulting learning signal is dominated by the gradient norm ($\rho \|\nabla L(w)\|$) rather than the curvature.
• The Consequence: Because the gradient norm dominates the signal, SAM's effectiveness is largely deferred to the late stages of training when gradients naturally become small enough for curvature terms to become comparable. This creates a discrepancy between the training mechanism and the goal of finding flat minima throughout the entire optimization process.

2. Methodology: Loss-Equated SAM (LE-SAM)

To resolve this mismatch, the authors propose Loss-Equated SAM (LE-SAM), a mechanism-level reformulation that inverts the traditional SAM paradigm. Instead of fixing the perturbation radius in parameter space, LE-SAM fixes a loss-space budget $\sigma$.

Core Mechanism

Inspired by the classical Polyak step-size, LE-SAM scalarizes the first-order loss variation.

1. Fixed Budget: Instead of fixing $\rho$, the algorithm fixes a target loss increase $\sigma$ in the loss space.
2. Back-Solving Radius: The perturbation radius $\rho_t$ is dynamically calculated to ensure the first-order loss variation matches the budget $\sigma$:
   $$\rho_t = \frac{\sigma}{\|\nabla L(w)\|_2 + \varrho}$$
   where $\varrho$ is a small constant for numerical stability.
3. Surrogate Loss: When substituting this adaptive radius back into the Taylor expansion of the loss, the first-order term becomes a constant scalar $\sigma$:
   $$L(w + \epsilon_t) \approx L(w) + \underbrace{\sigma}_{\text{scalar constant}} + \frac{1}{2} \epsilon_t^\top H(w) \epsilon_t$$
   Since $\nabla_w \sigma = 0$, the gradient-norm penalty is effectively removed from the optimization signal. The learning signal is shifted entirely toward the second-order curvature term ($\frac{1}{2} \epsilon_t^\top H(w) \epsilon_t$).

Stabilization Strategies

To prevent instability when gradient norms are small (which could lead to excessively large $\rho_t$), LE-SAM employs two safeguards:

• Radius Clipping: A maximum radius $\rho_{max}$ is enforced.
• Loss Budget Annealing: The budget $\sigma$ is gradually annealed to 0 during the final stages of training (e.g., via cosine decay) to ensure smooth convergence.

3. Key Contributions

The paper claims the following contributions:

1. Identification of Objective Mismatch: The authors demonstrate that the adversarial perturbation in standard SAM is generated via a first-order linearized surrogate under a fixed-radius constraint, causing the learning signal to be dominated by gradient magnitude rather than the second-order curvature required for flat minima.
2. Proposition of LE-SAM: A mechanism-level reformulation that fixes the loss budget instead of the radius. This scalarizes the first-order variation, removing gradient-norm dominance and shifting the optimization signal toward curvature-dominated terms throughout the entire training process.
3. Empirical Superiority: Extensive experiments show that LE-SAM consistently outperforms standard SAM and its variants (including ASAM, ESAM, F-SAM, and Eigen-SAM) across diverse benchmarks, achieving state-of-the-art (SOTA) performance.

4. Experimental Results

The authors evaluated LE-SAM on multiple tasks and datasets:

• Training from Scratch:
  • CIFAR-10/100: Using ResNet-18, WideResNet-28-10, and PyramidNet-110, LE-SAM achieved the highest accuracy among all SAM variants. On CIFAR-100, it outperformed standard SAM by 1.34 to 1.49 points.
  • ImageNet: Training ResNet-50 and ResNet-101 from scratch, LE-SAM outperformed SAM by 0.71 and 0.77 points respectively.
• Transfer Learning: Fine-tuning a pre-trained ViT-B-16 on CIFAR-10/100, LE-SAM consistently surpassed SAM.
• Domain Generalization (SDG): On PACS and VLCS datasets, LE-SAM improved average accuracy by 2.18 and 0.84 points over SAM, respectively.
• Robustness: Under $L_2$ and $L_\infty$ PGD adversarial attacks, LE-SAM demonstrated stronger robustness, outperforming SAM by up to 2.47 points.
• Long-Tailed Learning: On CIFAR-100-LT, combining LE-SAM with the SOTA method GLMC yielded further improvements over GLMC+SAM.

Mechanism Analysis

• Flatness Metrics: Hessian trace and top eigenvalue analysis confirmed that LE-SAM converges to flatter minima (lower Hessian trace and max eigenvalue) compared to SAM and SGD.
• Training Dynamics: Unlike SAM, which shows significant gains only in the late training stage, LE-SAM demonstrated substantial performance gaps throughout the entire training process, validating that the curvature signal is active from the beginning.
• Computational Cost: LE-SAM maintains the same asymptotic complexity as SAM (two forward-backward passes per iteration) with negligible overhead for calculating the dynamic radius.

5. Significance and Claims

The paper argues that the primary limitation of existing Sharpness-Aware Minimization methods is their reliance on a fixed-radius, first-order approximation, which inadvertently prioritizes gradient magnitude over curvature. By "fixing the loss, not the radius," LE-SAM fundamentally rethinks the adversarial perturbation mechanism.

The authors claim that this approach provides a unified view of sharpness-aware learning that is intrinsically curvature-aware. By decoupling the perturbation size from the gradient norm, LE-SAM offers a more direct and robust path to flat regions, resulting in superior generalization performance across various architectures, datasets, and training scenarios (including transfer learning and domain generalization) without incurring significant computational overhead.