Technical Summary: Grokking or Glitching? How Low Precision Drives Slingshot Loss Spikes

1. Problem Statement

Deep neural networks trained without explicit regularization frequently exhibit a phenomenon known as "Grokking": models achieve sudden generalization long after reaching perfect training accuracy. This process is often accompanied by the Slingshot Mechanism: periodic instabilities characterized by rapid, near-exponential growth in the norm of the final layer parameters, followed by abrupt spikes in training loss.

While existing literature attributes these spikes to intrinsic optimization dynamics—such as the "Edge of Stability" (EOS) or interactions between gradient scales and adaptive optimizers—the precise triggering mechanism remains unclear. This paper investigates whether these instabilities are inherent to the optimization landscape or artifacts of finite-precision arithmetic.

2. Methodology and Theoretical Framework

The authors postulate that the Slingshot Mechanism is not an intrinsic optimization phenomenon, but rather the result of limitations in floating-point arithmetic during the computation of Cross-Entropy (CE) loss.

2.1 Softmax Collapse (SC)

The core of the mechanism lies in the absorption error inherent to floating-point addition. During training, and when the model reaches a high-confidence regime (perfect training accuracy), the logit for the correct class ($z_r$) becomes significantly larger than the logits for incorrect classes ($z_k$).

• In the Softmax calculation, the denominator contains a Log-Sum-Exp operation: $Z = \log(\sum \exp(z_k))$.
• When the margin $z_r - \max_{k \neq r} z_k$ exceeds a threshold determined by mantissa precision (e.g., $\approx 16$ for float32), the smaller terms are absorbed into the larger term due to limited precision.
• Consequently, the calculated probability for the correct class becomes exactly 1, and the gradient with respect to the correct class logit, $\partial L / \partial z_r$, is rounded to exactly zero. This phenomenon is termed Softmax Collapse (SC).

2.2 Breaking the Zero-Sum Condition

In exact arithmetic, the sum of gradients over all classes for a single sample is zero ($\sum_k \nabla_{W_k} L = 0$), ensuring that the global mean of the classifier weights ($W_G$) remains static.

• Under SC, the gradient for the correct class vanishes, while gradients for incorrect classes remain non-zero (albeit small).
• This breaks the zero-sum condition and leads to a net drift in the update of the global classifier mean $W_G$. Specifically, $W_G$ drifts in the direction of $-\mu_G$, where $\mu_G$ is the global mean of the features from the penultimate layer.

2.3 Numerical Feature Inflation (NFI)

The paper identifies a positive feedback loop termed Numerical Feature Inflation (NFI):

1. The drift in $W_G$ alters the gradient flow back to the feature layer.
2. The feature mean $\mu_G$ acquires a drift in the direction of $-W_G$.
3. This creates a coupled dynamic where $W_G$ and $\mu_G$ become anti-parallel, and their norms grow exponentially.
4. As these norms grow, the margin for some samples (particularly outliers) eventually falls below the absorption threshold, causing the gradient for the correct class to abruptly reappear.

2.4 The Slingshot Spike

The reappearance of the correct class gradient triggers the loss spike:

• Prior to the spike, the gradient for the correct class is zero, and gradients for incorrect classes are tiny. The second-moment accumulator ($v_t$) of the Adam optimizer decays, causing the effective learning rate ($\eta / \sqrt{v_t + \epsilon}$) to become extremely large.
• When the gradient suddenly reappears, this massive effective learning rate leads to a disproportionately large parameter update, driving the loss back to the level of random guessing (the "Slingshot").

3. Main Contributions

1. Theoretical Explanation of Slingshot: The paper provides a rigorous proof that the Slingshot Mechanism is a numerical artifact caused by computing CE loss with finite precision, specifically linking it to Softmax Collapse and the subsequent breaking of the zero-sum gradient condition.
2. Identification of NFI: The authors define Numerical Feature Inflation (NFI), a feedback loop between the global classifier mean and the global feature mean that drives the exponential growth of parameter norms observed prior to loss spikes.
3. Distinction from Intrinsic Dynamics: The work demonstrates that Slingshot is distinguishable from the "Edge of Stability" (EOS). While EOS involves oscillations due to Hessian curvature, Slingshot occurs in regimes where Hessian eigenvalues vanish (interpolation regime) and is eliminated by increasing precision.
4. Practical Interventions: The paper proposes and validates methods to suppress NFI, including:
   • Using float64 for loss calculation (eliminating absorption errors).
   • Enforcing the zero-sum condition via gradient projection.
   • Applying Batch Normalization immediately before the final classifier to remove the global mean component.
   • Increasing the Adam $\epsilon$ parameter to limit the effective learning rate.

4. Experimental Results

The authors validate their theory across multiple architectures (MLP, VGG11, ResNet18, ViT, Transformer) and datasets (Modular Arithmetic, CIFAR-10, FineWeb for LLMs).

• Precision Sensitivity: Switching loss calculation to float64 completely eliminates Slingshot spikes, even when model parameters remain in float32. This confirms that the instability stems from loss calculation, not model dynamics.
• Architecture Dependence: The phenomenon is observed in most architectures but notably absent in ResNet18. The authors attribute this to ResNet18's ability to reduce the residual probability mass ($\epsilon$) quickly enough to prevent the NFI feedback loop from entering the exponential growth regime before the model stabilizes.
• Mitigation:
  • Layer Normalization (LN) applied before the classifier fails to mitigate the effect and can accelerate spikes by constraining feature norms.
  • Batch Normalization (BN) applied immediately before the final linear layer successfully eliminates spikes by centering features and removing the drift component.
  • Label Smoothing eliminates precision-induced spikes but introduces a different instability related to the "Edge of Stability" due to non-vanishing Hessian eigenvalues.
• LLM Context: In Large Language Model training, Slingshot spikes are rare in full-batch settings due to mini-batch stochasticity; however, the NFI mechanism still drives anomalous logit growth. The authors note that in language modeling, imbalances in token frequency (Zipf's Law) can generate a large global mean $W_G$ even without numerical collapse, leading to a different dynamic where low precision can partially suppress frequency-induced divergence.

5. Significance and Claims

The paper reinterprets the Slingshot Mechanism not as mysterious optimization behavior, but as a numerical dynamic of finite-precision training.

• Reinterpretation of Gradient Flow: The results highlight a gap between theoretical gradient flow analyses (which assume infinite precision and often predict logarithmic norm growth) and practical training, where numerical errors can drive exponential growth.
• Stability in Low-Precision Training: As the field moves toward lower precision (BF16, FP8, FP4) for large-scale training, understanding NFI is critical. The authors argue that numerical errors in log-probability calculations represent a distinct failure mode that can destabilize training, separate from quantization errors in matrix multiplications or attention sinks.
• Modest Scope: The authors acknowledge limitations, noting that their analysis relies on the Unconstrained Feature Model assumption and primarily investigates the interaction between the final layer and the penultimate layer features. They do not claim to explain all forms of training instability, but specifically target the periodic loss spikes observed in unregularized, long-term training.

In summary, the paper claims that Slingshot is a glitch, not a feature, driven by floating-point absorption errors, and provides concrete, verifiable explanations and interventions for anomalous parameter growth in the late stages of training.