Technical Summary: Simplicity Suffices for Parameter Noise Injection in Stochastic Gradient Descent

Problem Statement

Injecting noise into the optimization process is a well-established technique for enhancing the training and generalization of deep neural networks, often interpreted as a smoothing mechanism to help escape sharp minima and saddle points. However, despite a broad landscape of existing approaches—ranging from noise injection into parameters, gradients, or via gradient-free schemes—it remains unclear which specific design choices yield meaningful practical gains.

Two primary questions drive this investigation:

1. Batch Efficiency: How can one efficiently pair each training example in a mini-batch with its own unique perturbation without breaking the computational efficiency of batched matrix operations?
2. Complexity vs. Benefit: Do sophisticated noise parameterizations (e.g., diagonal Gaussian distributions with time-varying variances) or multi-sample gradient averaging (using multiple perturbed forward passes per update) provide significant improvements over simpler alternatives, such as isotropic noise with a single perturbation?

Methodology

Efficient Noise Injection (ENSGD)

The authors address the first challenge by leveraging a distributional identity for linear layers. Standard noise injection applies a single random weight perturbation $Z$ to an entire batch, meaning every example $x_i$ receives the same noise. To assign unique perturbations while maintaining batched computation, the authors propose injecting noise into the output of the matrix product rather than the weights themselves.

Given a linear layer $L_k(X) = \sigma(XW_k)$, if the weights are perturbed by $Z \sim \mathcal{N}(0, \sigma^2 I)$, the result for a single input $x$ is $x(W_k + Z) = xW_k + xZ$. By applying Lemma 1, which states that $xZ \sim \mathcal{N}(0, \sigma^2 \|x\|^2 I)$, the authors derive Theorem 1. This theorem proves that unique perturbations for every example in a batch can be simulated by adding a noise term $\psi$ to the pre-activation output, where $\psi_i \sim \mathcal{N}(0, \sigma^2 \|x_i\|^2 I)$.

This leads to the Efficient Noisy Stochastic Gradient Descent (ENSGD) algorithm (Algorithm 2), which:

• Assigns a unique perturbation to each example in a mini-batch.
• Preserves the efficiency of a single matrix product per forward pass.
• Can be generalized to include biases and other linear layers.

Experimental Design

The authors systematically evaluate the proposed method against several baselines on three datasets:

• CIFAR-100: Image classification using a CNN.
• AG News & IMDB: Text classification using TF-IDF vectors and fully-connected MLPs.

The experiments vary three key factors:

1. Noise Magnitude ($\sigma$): Tested across a log-uniform range.
2. Number of Samples ($s$): Comparing single-sample updates ($s=1$) against multi-sample averaging ($s \in \{2, 4, \dots, 128\}$).
3. Noise Parameterization: Comparing an Isotropic baseline (constant variance) against:
   • Moving: Exponential moving average of empirical gradient variance.
   • SqGrad: Scaled by Adam's second moment.
   • InvSqGrad: Scaled by the inverse of Adam's second moment.

Key Results

1. Efficiency of Unique Perturbations

Comparing the standard Noisy SGD (NSGD, Algorithm 1) with the proposed ENSGD (Algorithm 2) on CIFAR-100, ENSGD consistently achieved lower test error across all tested noise levels. This confirms that assigning unique perturbations to each example within a batch improves optimization performance without sacrificing computational efficiency.

2. Diminishing Returns of Multi-Sample Averaging

Experiments varying the number of noise samples $s$ per update step revealed that increasing $s$ beyond 1 yields diminishing returns.

• The best test error was achieved with $s=1$ (0.6185).
• Increasing $s$ to 128 resulted in a slight degradation (0.6261) while significantly increasing computational cost.
• This suggests that a single perturbed forward pass per update step captures the vast majority of the benefits of noise injection.

3. Simplicity of Noise Parameterization

Across CIFAR-100, AG News, and IMDB, the simple Isotropic noise parameterization performed competitively against more complex schemes:

• Image Classification (CIFAR-100): The Isotropic method (0.6241) was only marginally worse than the best-performing method, SqGrad (0.6213).
• Text Classification: The gains from noise injection were modest (at most 0.11 percentage points) compared to the baseline, likely due to the already smooth optimization landscape of TF-IDF representations. In these settings, the Isotropic method matched or closely approximated the best results.
• Complexity: Sophisticated methods like SqGrad and InvSqGrad did not provide systematic advantages. Notably, SqGrad failed to improve over the baseline on IMDB at any tested noise level.
• Stability: No method caused significant degradation at its optimal noise level.

Significance and Claims

The paper concludes that simplicity suffices for parameter noise injection in Stochastic Gradient Descent. The authors claim that practitioners do not need to resort to elaborate perturbation designs, such as diagonal Gaussian parameterizations with time-varying variances or multi-sample gradient averaging, to reap the optimization and generalization benefits of noisy SGD.

Instead, the combination of:

1. Isotropic noise (constant variance), and
2. A single perturbed forward pass per update step (enabled by the efficient ENSGD algorithm),

is sufficient to recover most of the benefits of more complex schemes. This finding simplifies the implementation of noise-based regularization, suggesting that lightweight strategies are robust across diverse task settings, including those where the underlying optimization landscape is already relatively smooth.

Future Work

The authors note plans to broaden the experimental section with additional datasets to further consolidate these findings and to extend the noise injection method from linear layers to convolutional layers (involving weight sharing).