🤖 machine learning

Bug or Feature $^2$ : Weight Drift, Activation Sparsity, and Spikes

This paper identifies a fundamental negative weight drift caused by the interaction between standard losses and positively biased activations, which induces high activation sparsity and accuracy cliffs in deep networks, leading the authors to propose clipped ReLU $^2$ and GELU $^2$ as effective solutions that balance sparsity, stability, and performance.

Original authors: Egor Shvetsov, Aleksandr Serkov, Shokorov Viacheslav, Redko Dmitry, Vladislav Goloshchapov, Evgeny Burnaev

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Egor Shvetsov, Aleksandr Serkov, Shokorov Viacheslav, Redko Dmitry, Vladislav Goloshchapov, Evgeny Burnaev

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a massive, complex machine made of thousands of tiny switches (neurons) that learn to recognize patterns. For years, engineers have been tweaking these machines by trial and error, adding features that seem to work well without fully understanding why they work.

This paper, titled "Bug or Feature2," investigates a hidden quirk in how these machines learn. The authors discovered a "ghost" in the machine: a tendency for the internal settings (weights) to slowly drift in a negative direction, which causes many of the switches to turn off completely.

Here is the breakdown of their findings using simple analogies:

1. The "Negative Drift" (The Tug-of-War)

Imagine the machine starts with all its settings balanced perfectly around zero. The authors found that because of how the machine calculates its mistakes (using standard math formulas called "loss functions") and how it processes information (using "activation functions" like ReLU), there is a subtle, invisible tug-of-war.

The Mechanism: In the very first few seconds of training, the math pushes the settings slightly toward the negative side.
The Result: Once a setting goes negative, it stays negative. It's like a ball rolling down a hill; once it starts, it keeps going until it hits a wall. This happens even if the machine is fed random, nonsense data. It's a flaw in the training process itself, not the data.

2. The "Silent Switches" (Activation Sparsity)

Now, imagine those switches are lightbulbs.

With ReLU (a common switch type): If the setting drifts negative, the lightbulb turns off completely and stays off. The authors found that in some models (like a small language model called GPT-nano), up to 90% of the lightbulbs turn off and stay silent.
The Question: Is this a Bug or a Feature?
- Bug: If too many lights go out, the machine might go blind and stop learning.
- Feature: If the machine can do its job with fewer lights on, it might be more efficient.

3. The "Cliff" (The Danger Zone)

The researchers tested how much silence the machine could handle before it broke. They found a sharp "Cliff."

The Safe Zone: If you turn off up to 70% of the switches, the machine still works almost as well as before. It's surprisingly robust.
The Cliff: If you try to turn off more than 70% (say, 80% or 90%), the machine's performance crashes. It's like trying to drive a car with only one wheel; it works fine until you hit a certain speed, then it falls apart.
The Exception: Machines with "skip connections" (like ResNet or Transformers) are like cars with backup wheels; they can handle much more silence before crashing.

4. The "Squaring" Experiment (ReLU2)

The authors tried a new type of switch called ReLU2, which squares the signal (makes big numbers even bigger).

The Problem: This created "Spikes." Imagine a few lightbulbs suddenly flashing so brightly they burn out or blind the system. In the middle layers of the machine, these spikes became dangerously large.
The Fix: They found that clipping the spikes (putting a ceiling on how bright the bulb can get) solved the problem.
The Winner: A "Clipped ReLU2" worked better than the original, and a "Clipped GELU2" (a different smooth switch) actually performed the best on the language model, achieving the lowest error rate.

5. The "Freeze" Trick (Computational Efficiency)

The "drift" happens mostly in the first few steps of training. After that, the settings stabilize.

The Idea: Usually, the machine recalculates its "center of gravity" (normalization statistics) every single time it sees a new image or sentence. This is slow.
The Hack: The authors found that once the initial drift settles (after a short "warm-up"), you can freeze those calculations. You stop recalculating and just use the numbers from the start.
The Benefit: This makes the machine train about 25–30% faster without hurting its final intelligence. It's like setting the thermostat once the room reaches the right temperature, rather than checking the temperature every second.

Summary

The paper reveals that modern AI models naturally develop "dead zones" where many neurons stop working due to a mathematical quirk in how they learn.

It's not a bug in the data; it's a bug in the optimization math.
Silence is okay up to a point (70%), but too much silence causes a crash.
New switches (Squared functions) can be powerful but need "clipping" to prevent dangerous spikes.
Freezing early calculations can make training significantly faster.

The authors conclude that what looks like a random side effect is actually a controllable mechanism that, if understood, can help us build faster and more efficient AI.

Technical Summary: Bug or Feature2: Weight Drift, Activation Sparsity and Spikes

Problem Statement

Modern neural architecture design has evolved through incremental empirical choices, yet the intrinsic mechanisms governing training dynamics remain partially understood. This paper identifies a systematic negative weight drift induced by the interaction between standard loss functions (MSE, cross-entropy) and positively biased activation functions (e.g., ReLU, GELU, SiLU) in the absence of mean-centering normalization.

The core problem is that during the early iterations of training, gradient descent drives weights toward negative values. This occurs because the expected gradient with respect to positive pre-activations is non-negative. Consequently, pre-activations shift downward, causing more neurons to fall below zero. When passed through activation functions like ReLU, these negative pre-activations are squashed to zero, inducing hard activation sparsity. The authors investigate whether this emergent sparsity is a beneficial feature for efficiency or a detrimental bug that degrades model performance. Additionally, the paper examines pathological activation spikes observed in squared activation functions (e.g., ReLU2) within transformer architectures.

Methodology

1. Formal Analysis of Weight Drift

The authors provide a formal proof for ReLU-based Multi-Layer Perceptrons (MLPs) at initialization.

Theoretical Framework: They define an effective weight matrix $V_{eff}$ representing the composition of subsequent linear layers and ReLU gates.
Key Theorem: Under zero-mean i.i.d. weight initialization, the rows of $V_{eff}$ exhibit zero mean but non-negative cross-correlation ( $E[\langle v_i, v_j \rangle] \ge 0$ ).
Gradient Analysis: For both MSE and cross-entropy losses, the expected gradient with respect to any positive pre-activation is non-negative. Since gradient descent updates weights in the negative direction of the gradient ( $w \leftarrow w - \eta \nabla w$ ), this consistently positive gradient drives weights toward negative values.
Self-Reinforcing Cycle: As weights become negative, pre-activations shift further below zero, increasing the fraction of dead neurons and reinforcing the drift.

2. Empirical Validation

The authors verify these dynamics across diverse settings:

Architectures: MLP, ResNet-18, MaxViT, MP-SENet, and GPT-nano.
Activations: ReLU, GELU, SiLU, NoisyReLU, SUGARBSiLU, and ReLU2.
Data: Training on both real-world datasets (CIFAR-10, ImageNet-1K, FineWeb) and entirely random inputs to isolate optimization dynamics from data distribution.
Optimizers: SGD, SGD with momentum, and Adam/AdamW.

3. Sparsity Control and Trade-off Analysis

To determine if sparsity is a "bug" or "feature," the authors explicitly control sparsity levels using two mechanisms:

Top-K Sparsity: Retains the top $k\%$ of activations (used with GELU).
Percentile Centering (PC): A proposed method that shifts activations by a target percentile $q$ rather than the mean before applying ReLU, directly controlling the fraction of zeroed activations.
Evaluation: They benchmark 79 configurations across architectures to map the relationship between sparsity levels and model accuracy/loss.

4. Investigation of Squared Activations and Spikes

The paper analyzes ReLU2 ( $max(0, x)^2$ ) and GELU2 in GPT-nano.

Spike Detection: They observe massive activation spikes in layers 2–4 of GPT-nano when using ReLU2.
Mitigation: They test clipping these activations at thresholds (15 and 50) to determine if the spikes are harmful (bugs) or informative (features).

Key Results

1. Negative Weight Drift is Intrinsic

Universality: The drift occurs across all tested architectures and activation functions, including those without explicit normalization (e.g., RMSNorm).
Optimization Origin: The phenomenon persists even on random data, confirming it is a property of the optimization dynamics rather than the data distribution.
Magnitude: The drift is most pronounced in the first training iterations and stabilizes as gradients diminish. Momentum-based optimizers accelerate this initial drift.
Sparsity Consequence: In ReLU-based models, this drift induces natural sparsity reaching up to 90% in GPT-nano without explicit regularization.

2. The Sparsity-Accuracy "Cliff"

Trade-off Curve: Performance remains stable across moderate sparsity levels ( $s \lesssim 0.7$ ) but degrades sharply beyond a critical threshold (the "cliff").
Architecture Dependence:
- MLPs: Suffer catastrophic collapse at ~85% sparsity.
- ResNet: Shows monotonic degradation with structured (channel-wise) sparsity but retains robustness with unstructured sparsity.
- Transformers (ViT, GPT): Exhibit extreme resilience. GPT-nano maintains near-baseline performance up to ~91% sparsity.
Mechanism Independence: The specific method of inducing sparsity (Top-K vs. Percentile Centering) has a marginal impact compared to the sparsity level itself.

3. Activation Functions and Sparsity

GELU: Achieves the highest accuracy in most classification tasks but induces minimal natural sparsity.
ReLU2: Highly sensitive to normalization. It collapses on MLPs without normalization but achieves the lowest validation loss on GPT-nano (3.250) compared to GELU (3.260).
ReLUfication: Converting GELU-trained models to ReLU via one epoch of fine-tuning recovers 55–74% sparsity with negligible accuracy loss (<1%), offering a practical strategy for efficiency.

4. Pathological Spikes and Clipping

Spike Origin: Spikes in GPT-nano originate in the up-projection of MLP blocks and are amplified by the squaring operation in ReLU2.
Clipping Benefits:
- Clipped ReLU2: Outperforms unclipped ReLU2 (3.236 vs. 3.251), indicating that extreme spikes are harmful.
- Clipped GELU2: Achieves the best overall validation loss (3.233) on GPT-nano.
- ViT Results: Squared activations do not outperform GELU on MaxViT, suggesting the benefits of squared activations may be specific to autoregressive models.

5. Computational Efficiency via Accumulation Stop

Observation: Since weight drift stabilizes after a few thousand steps, dynamic computation of normalization statistics (percentiles, means) is unnecessary for the entire training duration.
Method: The authors propose Accumulation Stop (AS), where statistics are computed via an Exponential Moving Average (EMA) during a warm-up phase (e.g., 50k steps) and then frozen.
Result: This recovers training throughput to near-baseline levels (removing the ~25–30% overhead of dynamic percentile computation) without degrading model quality (FID/IS scores remain stable or improve).

Significance and Claims

The paper reframes activation sparsity not as an emergent property of data, but as a controllable consequence of the interaction between loss functions, activation functions, and normalization layers.

Mechanistic Understanding: The work provides a theoretical and empirical explanation for why modern architectures (which often omit mean-centering, e.g., using RMSNorm) exhibit systematic weight drift and high sparsity.
Design Guidelines:
- Sparsity Control: Aggressive sparsity (up to ~70-80%) can be achieved without significant performance degradation in transformer-based models, offering a pathway for computational efficiency.
- Activation Choice: While ReLU2 offers performance gains in LLMs, it requires clipping to mitigate pathological spikes. GELU remains a robust general-purpose baseline.
- Efficiency: The "Accumulation Stop" technique offers a practical method to reduce training overhead by freezing normalization statistics after the initial drift stabilization phase.
Limitations: The authors note that theoretical proofs strictly apply to initialization and ReLU, with extensions to smooth activations being empirical. The applicability of these findings to frontier-scale LLMs (1B+ parameters) and the optimal clipping thresholds for different architectures remain open questions.

The paper concludes that understanding these "seemingly incidental" design choices is crucial for future architecture development, particularly as the field moves toward non-centering normalization and sparse inference.