📊 statistics

Numerical stability analysis of large language models

This paper presents a mixed-precision analysis of transformer inference that derives novel error bounds and stability conditions for key components like LayerNorm, RMSNorm, and self-attention, ultimately revealing that numerical stability is governed by the interplay between weight magnitudes and residual stream growth, a finding validated by experiments on GPT-2.

Original authors: Stanislav Budzinskiy, Wenyi Fang, Longbin Zeng, Philipp Petersen

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Stanislav Budzinskiy, Wenyi Fang, Longbin Zeng, Philipp Petersen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (like the ones powering chatbots) as a massive, multi-story factory. Inside this factory, information flows from the ground floor to the top floor, passing through hundreds of rooms (layers). In each room, the information gets processed, mixed, and reshaped before being passed to the next.

The paper you provided is a safety inspection report for this factory. The authors are asking: "If we start with a tiny, almost invisible speck of dust (a tiny computer rounding error) in the first room, how big will that speck be by the time it reaches the top?"

Here is a breakdown of their findings using simple analogies:

1. The "Low-Precision" Shortcut

To make these factories run fast and cheap, engineers often use "low-precision" math. Think of this like measuring ingredients with a ruler that only has inch marks instead of millimeter marks. It's faster, but you lose a tiny bit of accuracy.

The Problem: When you do thousands of calculations in a row, those tiny "inch mark" errors can pile up. The authors wanted to know: Does the factory collapse under the weight of these tiny errors, or does it stay stable?

2. The "Normalizing" Doors (LayerNorm vs. RMSNorm)

Before information enters a new room, it has to pass through a "normalizing door" that adjusts the size of the data.

The Old Door (LayerNorm): This door subtracts the average size of the data. The authors found that if the data has one massive outlier (a single huge number that is way bigger than the rest), this door can get shaky. It's like trying to balance a seesaw where one side has an elephant and the other has a mouse; the math gets sensitive.
The New Door (RMSNorm): This is the modern door used in top models. It doesn't subtract the average; it just scales based on the total size. The authors found this door is unconditionally stable. Even with that elephant on the seesaw, the door holds firm. It's a more robust design.

3. The "Attention" Spotlight

The most famous part of these models is "Self-Attention," which decides which words in a sentence are important to each other.

The "Softmax" Switch: This is the mechanism that turns raw scores into probabilities (percentages). The authors analyzed a common trick called "shifting" (subtracting the biggest number before calculating) to prevent the math from exploding (overflowing).
The Finding: They proved that this "shifting" trick is safe. It might make the system slightly more sensitive to noise (at most 4 times more), but it doesn't break the math. It's like wearing sunglasses to look at the sun; it changes how you see things slightly, but it doesn't blind you.
The "Attention Sink": They also noticed that in long conversations, the model often focuses heavily on the very first few words (the "sink"). Their math shows that even with these long sequences, the errors don't spiral out of control as badly as old theories suggested.

4. The "Residual Stream" (The Conveyor Belt)

The data flows through the factory on a "residual stream"—a conveyor belt that carries the main message forward.

The Growth: As the data moves up the floors, the conveyor belt naturally gets "heavier" (the numbers get bigger).
The Weights: The machines on the conveyor belt (the weights) are scaled down as the factory gets taller to keep things balanced.
The Big Discovery: The authors found a golden rule: If you scale the conveyor belt up or down, the relative error stays the same.
- Analogy: Imagine you are walking on a treadmill. If you double the speed of the treadmill and also double the size of your shoes, your walking style (relative to your size) doesn't change.
- The Catch: This only works if the conveyor belt behaves "linearly" (predictably). If you scale the weights too much, the conveyor belt might suddenly switch to a completely different mode of operation (a "qualitative transition"). In that specific case, the stability breaks. But as long as the belt stays in its normal rhythm, scaling the weights doesn't hurt the accuracy.

5. The "GPT-2" Experiment

To prove their math wasn't just theory, they tested it on a real model called GPT-2.

Random Weights: When they used a model with random, untrained weights, the errors grew slowly and predictably.
Trained Weights: When they used a real, trained model, the errors grew a bit faster (exponentially), but still within manageable limits.
The Verdict: The math held up. The "relative error" (the mistake compared to the size of the data) stayed consistent unless they forced the model into a weird, unstable state by changing the weights too drastically.

Summary

The paper concludes that Large Language Models are numerically stable.
Even though they use "rough" math (low precision) and have to handle massive numbers, the architecture is designed in a way that keeps the errors from piling up into a disaster. The key to this stability is the interplay between the size of the weights and the growth of the data stream. As long as the model doesn't get pushed into a weird, unstable state, the "noise" introduced by low-precision math remains a tiny, manageable speck of dust, not a landslide.

Technical Summary: Numerical Stability Analysis of Large Language Models

Problem Statement

Large Language Models (LLMs), built upon Transformer architectures, rely heavily on low-precision arithmetic for efficient training and inference. While quantization and mixed-precision strategies offer significant computational gains, they introduce numerical perturbations that propagate through the network. These errors are compounded by the inherent rounding errors of floating-point (FP) operations. Although empirical studies have extensively documented the impact of quantization, rigorous theoretical analyses of numerical stability in deep Transformer architectures remain scarce. This work addresses this deficit by developing a deterministic, mixed-precision numerical analysis of Transformer inference, aiming to derive bounds for condition numbers and forward errors of the architecture's constituent parts.

Methodology

The authors employ a deterministic rounding error analysis framework (Higham, 2002) adapted for mixed-precision environments. The analysis distinguishes between four distinct precisions:

Quantisation precision ( $u_q$ ): Used for rounding operands in General Matrix Multiplication (GEMM).
Accumulation precision ( $u_a$ ): Used for summing inner products.
Working precision ( $u_w$ ): Used for evaluating non-linear functions.
Residual precision ( $u_r$ ): Used for storing state variables in the residual stream.

The study focuses on a representative decoder-only Transformer architecture featuring pre-normalization, causal multi-head self-attention with Rotary Positional Embeddings (RoPE), and either GELU or SwiGLU feedforward (FF) mechanisms. The analysis proceeds by:

Deriving condition numbers (normwise, componentwise, and mixed) for individual components: LayerNorm/RMSNorm, FF blocks, Softmax, and Self-Attention.
Establishing forward error bounds for these components under FP arithmetic, accounting for specific accumulation algorithms (e.g., recursive vs. pairwise summation) and absorption effects in Softmax.
Aggregating these local bounds to analyze the global forward error of the deep Transformer architecture, explicitly modeling the growth dynamics of the residual stream.

Key Contributions

1. Normalization Stability (LayerNorm vs. RMSNorm)

The paper provides a comparative numerical stability analysis of LayerNorm and RMSNorm, specifically in the regime of "massive outliers" characteristic of modern LLM inference.

Findings: Neither function is strictly superior in terms of sensitivity (condition numbers). However, the forward error of LayerNorm is governed by the magnitude of the outlier's background entries, whereas RMSNorm exhibits unconditional forward stability.
Implication: The transition to RMSNorm in modern LLMs reduces computational complexity without strictly degrading numerical stability, though the specific error behavior depends on the input distribution.

2. Feedforward Mechanism Stability

The authors analyze the forward stability of FF blocks using smooth ReLU-like activations (e.g., GELU) and gated mechanisms (e.g., SwiGLU).

Findings: The FP evaluation of both GELU and SwiGLU mechanisms is proven to be forward stable. The error bounds are dominated by the quantization and accumulation errors of the underlying GEMMs rather than the non-linearities themselves.

3. Softmax Sensitivity and Attention Sinks

The analysis addresses the numerical behavior of Softmax, particularly in the presence of "attention sinks" (where probability mass concentrates on early tokens).

Shifted Evaluation: The paper confirms that while the shifted evaluation algorithm (subtracting the maximum to prevent overflow) prevents numerical instability, it increases sensitivity to perturbations by at most a factor of 4 compared to unshifted evaluation.
Concentrated Distributions: For concentrated probability distributions (common in attention sinks), the authors derive a novel rounding-error bound that accounts for absorption effects. This bound scales with the effective sequence length (the number of non-absorbed terms) rather than the total sequence length, explaining the empirical success of long-context inference.

4. Sequence-Length-Independent Lipschitz Bounds

A significant theoretical contribution is the derivation of novel bounds on the local Lipschitz constant of self-attention that are independent of the input sequence length ( $N$ ).

Improvement: These bounds improve upon previous mean-field bounds (Castin et al., 2024) in the discrete case. The bounds are strictly quadratic in the norm of the input, whereas previous discrete bounds contained exponential factors.
Outliers: The analysis shows that the presence of a massive outlier can further tighten these bounds, reducing the condition number of the attention head.

5. Global Architecture Stability and Residual Stream Dynamics

The paper aggregates local error bounds to analyze the entire deep Transformer.

Key Insight: The numerical stability of inference is determined by the interplay between weight magnitude and the growth dynamics of the residual stream.
Scaling Laws: The analysis validates that the standard depth-dependent scaling of residual-projection weights (introduced in GPT-2) preserves the propagation of relative rounding error, provided the scaling does not force a qualitative transition in the residual stream's dynamics. If the residual stream responds linearly to weight scaling, the relative error remains invariant.

Results and Experimental Validation

The theoretical bounds were validated using experiments with the GPT-2 XL model, comparing randomly initialized weights against pre-trained weights.

Error Growth: For randomly initialized models, the empirical error growth was sub-exponential ( $O(e^{l^{0.24}})$ ), consistent with the theoretical prediction of sub-exponential growth, though the empirical rate was milder than the worst-case bound.
Pre-trained Models: Pre-trained models exhibited slow exponential error growth. The discrepancy with polynomial-rate predictions is attributed to the variation in weight magnitudes with depth in pre-trained models, which violates the uniform scaling assumptions of the theorem.
Weight Scaling Experiments: Experiments scaling the residual-projection weights ( $W_O$ $W_{O}$ and $W_{down}$ $W_{d o w n}$ ) confirmed the theoretical hypothesis:
- In the randomly initialized model, the absolute rounding error scaled linearly with the weight scaling factor $\xi$ .
- In the pre-trained model, a "qualitative transition" occurred between scaling factors $\xi=2$ and $\xi=5$ , leading to a non-linear response in absolute error. Once the model settled into a new regime ( $\xi > 5$ ), the error returned to strictly proportional scaling.
- Crucially, in all cases, the relative rounding error remained invariant to the scaling, unless the scaling induced a bifurcation in the residual stream dynamics.

Significance

The paper establishes a rigorous theoretical foundation for understanding the numerical stability of Transformer inference. Its primary significance lies in:

Bridging Theory and Practice: It provides the first comprehensive deterministic analysis of mixed-precision errors in Transformers, moving beyond empirical heuristics.
Validating Architectural Choices: It mathematically justifies the use of specific scaling laws (e.g., GPT-2 initialization) and normalization techniques (RMSNorm) by showing they preserve relative error propagation under specific dynamic conditions.
Explaining Long-Context Success: The derived bounds on Softmax with absorption effects offer a theoretical explanation for why LLMs can handle long contexts despite the potential for error accumulation, as the effective number of contributing terms is often small due to attention sinks.
Defining Stability Limits: The work clarifies that numerical stability is not solely a function of precision or weight magnitude but is intrinsically linked to the dynamical behavior of the residual stream.

The authors conclude that while their worst-case bounds can be conservative (as seen in the comparison with GPT-2 XL pre-trained weights), the derived principles regarding the interplay of weight scaling and residual stream dynamics provide a robust framework for analyzing and designing numerically stable LLMs.