🤖 machine learning

A Geometric Analysis of Sign-Magnitude Asymmetry in a ReLU + RMSNorm Block under Ternary Quantization

This paper provides a geometric explanation for the robustness of pre-norm Transformers to ternary weight quantization, demonstrating that ReLU-induced directional asymmetry combined with RMSNorm's projection properties causes sign-flip perturbations to generate significantly more output energy than magnitude perturbations, while outlier features in real models account for deviations from this theoretical bound.

Original authors: Lei Dong

Published 2026-05-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Lei Dong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Why "Sign" Matters More Than "Size"

Imagine you are trying to understand a complex song. You could listen to the volume of every instrument (magnitude) or just the direction they are playing (sign: positive or negative).

For a long time, researchers thought you needed the volume to understand the music. But recent experiments showed something surprising: if you take a massive AI model (a Transformer) and throw away all the volume information, keeping only the direction (whether weights are +1, -1, or 0), the model still works almost perfectly.

This paper asks: Why does the direction matter so much more than the volume?

The authors provide a geometric explanation involving three main characters: Sign, ReLU (a gatekeeper), and RMSNorm (a filter).

The Three Characters and Their Roles

1. The Weight Vector (The Arrow)

Think of the AI's weights as a giant arrow pointing in a specific direction in a high-dimensional space.

The Discovery: The paper proves that the sign (the direction the arrow points) contains about 64% of the arrow's useful information.
The Analogy: Imagine a compass. Knowing the compass points "North" tells you a lot. Knowing the compass is "North" but also "slightly heavier" tells you almost nothing new. The paper shows that random changes in the "heaviness" (magnitude) are mostly noise, while changes in the "direction" (sign) are the real signal.

2. ReLU (The One-Way Gate)

ReLU is an activation function that acts like a gate. If the signal is positive, it lets it through. If it's negative, it blocks it (turns it to zero).

The Effect: This gate creates a "lopsided" world. It treats positive and negative signals differently.
The Analogy: Imagine a river with a dam that only lets water flow downstream if it's moving fast enough. If you push the water slightly backward (a sign flip), the dam stops it completely. If you just change how hard the water is pushing forward (magnitude), the dam still lets it through. This creates a hidden imbalance.

3. RMSNorm (The Direction Filter)

RMSNorm is a normalization step that forces all signals to have the same "length" (magnitude) but keeps their direction.

The Effect: It acts like a sieve that filters out anything pointing in the same direction as the main flow, but keeps anything pointing sideways (transverse).
The Analogy: Imagine a wind tunnel. If you throw a ball straight down the tunnel (radial), the wind tunnel absorbs the impact. If you throw a ball sideways (transverse), the wind tunnel lets it pass through and hit the target.

The Core Discovery: The "2.75x" Surprise

The paper's main mathematical result is a specific number: $\pi / (\pi - 2) \approx 2.75$ .

Here is what that means in plain English:
If you make a mistake in the AI's weights, there are two ways to do it:

Flip the Sign: Change a +1 to a -1.
Change the Magnitude: Change a +1 to a +0.5 (keeping the sign the same).

If you make these mistakes with the same "energy" (mathematically, the same Frobenius norm), the Sign Flip causes 2.75 times more damage to the AI's output than the Magnitude change.

Why?

Sign Flips: Because of the ReLU gate, flipping a sign often changes the direction of the signal drastically. When this hits the RMSNorm filter, the filter lets almost all of that damage through (because it's pointing sideways).
Magnitude Changes: Because the sign stays the same, the ReLU gate doesn't block the signal. The RMSNorm filter sees this change as "pointing in the right direction" and absorbs most of it, canceling out the damage.

The Verdict: The AI is much more sensitive to getting the direction wrong than getting the size wrong. This is why "Ternary Quantization" (keeping only signs and zeros) works so well.

The "Outlier" Twist: Why Real Models Are Even More Sensitive

The math above predicts a 2.75x difference. However, when the authors tested this on a real, massive AI model (TinyLlama), they found the damage was much higher (up to 1,000x in some cases).

Why the gap?
The math assumed the AI's internal signals were spread out evenly (like a smooth fog). But in real models, the signals are "spiky." A few specific neurons (called outliers) are incredibly loud and active, while most are quiet.

The Analogy: Imagine a choir where everyone sings at the same volume. If one person changes their pitch (sign flip), it's noticeable. But in a real AI, imagine one person is screaming at 100 decibels while the rest whisper. If you flip the sign of that screaming person, the whole choir sounds completely different.
The Finding: The paper shows that these "screaming" outliers amplify the damage of sign flips by a factor related to their loudness squared ( $n\alpha^2$ ). This explains why real models are so sensitive to sign errors, far beyond the basic 2.75x prediction.

What About Ternary Quantization? (The "Good" News)

The paper also explains why "Ternary Quantization" (using only -1, 0, +1) works.

When you quantize weights to -1, 0, or +1, you are essentially making a "Magnitude Change" (you are adjusting the size but keeping the sign the same).
Because the AI is naturally "blind" to magnitude changes (thanks to the RMSNorm filter absorbing them), this type of error is harmless.
The paper calculates that even with the ReLU gate flipping some signals, the error remains mostly "radial" (absorbed by the filter), making the AI very tolerant of this specific type of compression.

Summary of Key Takeaways

Direction is King: In modern AI architectures, the sign of a weight carries the most important information. The size is mostly noise.
The 2.75x Rule: In a simplified model, flipping a sign causes nearly 3 times more damage than changing the size.
The Filter Effect: RMSNorm acts like a filter that cancels out size errors but lets sign errors pass through.
The Outlier Effect: Real-world AI models have "loud" neurons. Flipping the sign of these loud neurons causes massive damage, explaining why real models are even more sensitive than the simple math predicts.
No Magic Multiplier: The authors tested if this damage compounds (multiplies) as the signal goes through many layers. It does not. The damage doesn't grow exponentially with depth; the "loudness" of outliers is the real reason for the high sensitivity.

In a nutshell: The paper proves that AI models are built like a house of cards where the direction of the cards matters, but their thickness doesn't. As long as you keep the direction right, you can make the cards very thin (or even just signs), and the house will still stand.

Technical Summary: A Geometric Analysis of Sign-Magnitude Asymmetry in a ReLU + RMSNorm Block under Ternary Quantization

1. Problem Statement

Recent advances in extremely low-bit Large Language Models (LLMs), specifically BitNet b1.58, demonstrate that pre-norm Transformer architectures can operate with weights constrained to the ternary set $\{-1, 0, +1\}$ while retaining significant linguistic capability. This phenomenon is remarkable because ternary quantization discards all magnitude information, preserving only sign patterns and sparsity. While empirical evidence (e.g., the Lottery Ticket Hypothesis) suggests weight signs are more critical than magnitudes, a rigorous theoretical explanation for why sign carries disproportionate functional information specifically in pre-norm (RMSNorm) architectures has been lacking. This paper seeks to provide a geometric explanation for this sign-magnitude asymmetry.

2. Methodology

The authors employ a combination of asymptotic geometric analysis, perturbation theory, and empirical validation on both synthetic models and real-world LLMs (TinyLlama-1.1B, Qwen2.5).

Theoretical Framework: The core analysis focuses on a two-layer model consisting of a linear layer, a ReLU activation, and an RMSNorm layer ( $\hat{y} = \text{RMSNorm}(W_2 \cdot \text{ReLU}(W_1 \hat{x}))$ ). The weights are assumed to be i.i.d. Gaussian, and inputs are delocalized (uniform on the sphere).
Perturbation Analysis: The study compares two types of weight perturbations with equal expected Frobenius norm:
1. Sign-flip: Randomly flipping the sign of entries (introducing a sign-altering error).
2. Magnitude (Sign-preserving): Adding noise that preserves the original sign of the weights (introducing a sign-preserving error).
Geometric Tools: The analysis utilizes the Fréchet derivative of RMSNorm (identified as a transverse projector), the Bussgang constant for sign quantization, and Price's theorem for ReLU correlations.
Experimental Validation: Experiments verify theoretical limits on synthetic data and test the compounding of effects across layers and the impact of "outlier features" (violations of the delocalization assumption) on real models.

3. Key Contributions and Results

3.1 The $\pi/(\pi-2)$ Asymmetry Constant

The paper's primary theoretical result (Theorem 3) establishes that in a two-layer ReLU + RMSNorm model with i.i.d. Gaussian weights, sign-flip perturbations produce approximately 2.75 times ( $\pi/(\pi-2)$ ) more transverse output energy than sign-preserving magnitude perturbations of equal norm, as the flip rate $p \to 0$ .

Mechanism: The asymmetry arises from a two-step geometric process:
1. ReLU creates directional asymmetry: The sign-preserving perturbation structure aligns with the weight vector via the Bussgang constant ( $\cos^2 \to 2/\pi$ ). ReLU propagates this alignment, resulting in a high "radial fraction" (energy aligned with the output direction) for magnitude perturbations ( $R_{mag} \approx 2/\pi$ ). Conversely, sign-flips produce a near-zero radial fraction ( $R_{sign} \approx p$ ).
2. RMSNorm exposes the asymmetry: RMSNorm acts as a radial filter (Theorem 2), projecting out components aligned with the output vector. It absorbs $\approx 63.7\%$ ( $2/\pi$ ) of the magnitude perturbation energy but preserves $\approx 99\%$ of the sign-flip energy (for small $p$ ). The ratio of surviving transverse energy is $\frac{1-p}{1-2/\pi} \approx \frac{\pi}{\pi-2}$ .

3.2 Ternary Quantization Error Characterization

The paper characterizes ternary quantization error (Theorem 4) as a sign-preserving perturbation.

The angular alignment between the original output and the quantized output converges to $\cos^2 \theta \to 2/\pi$ .
Crucially, the post-ReLU radial fraction of ternary error is calculated to be $\approx 0.365$ , which is within 0.4% of the pre-ReLU value ( $1 - 2/\pi$ ). This implies that ReLU is approximately transparent to the radial/transverse decomposition of ternary error, meaning the error behaves similarly to the idealized magnitude perturbation model regarding RMSNorm filtering.

3.3 Refutation of Multi-Layer Compounding

A natural hypothesis is that the single-layer 2.75× factor compounds multiplicatively across deep networks, explaining the extreme sensitivity of real models. The authors refute this (Section 7.1, Table 3).

Experiments on synthetic and trained models (TinyLlama) show that the transverse energy ratio decreases or plateaus with depth, rather than growing exponentially.
The large sign sensitivity observed in real models is not due to depth compounding.

3.4 The Role of Outlier Features

The paper identifies outlier features (heavy-tailed activation distributions violating the delocalization assumption) as the true source of the gap between the theoretical 2.75× baseline and the massive sensitivity seen in real models.

Leverage Theory: For an input dimension with amplitude $\alpha$ , a single sign-flip produces post-ReLU energy amplified by a factor $R \approx n\alpha^2$ relative to a delocalized entry (Appendix E).
Empirical Validation: On TinyLlama-1.1B, flipping signs in "outlier columns" (top 5% by activation magnitude) causes $\sim 1,265\times$ more perplexity damage than non-outlier columns. However, when accounting for the count of flips and the nonlinearity of Perplexity (PPL) vs. Negative Log-Likelihood (NLL), the leverage stabilizes at $\sim 10\times$ in the linear response regime ( $p \le 0.5\%$ ), matching the theoretical prediction $n \cdot E[\alpha^2]$ .

4. Significance and Claims

The paper claims to provide a geometric explanation for the robustness of pre-norm Transformers to ternary quantization:

Sign Dominance: Sign patterns capture $\approx 63.7\%$ of the directional energy of weight vectors (Fact 1), while magnitude with random signs carries vanishing directional information.
Architectural Synergy: The specific combination of Sign-preserving structure + ReLU + RMSNorm creates a mechanism where sign-altering errors are selectively amplified in the transverse (functional) direction, while sign-preserving errors (like ternary quantization) are largely absorbed by the radial filtering of RMSNorm.
Real-World Deviation: The extreme sensitivity of real models to sign flips is not an intrinsic property of depth, but a consequence of outlier features violating the delocalization assumptions required for the baseline theory.

5. Limitations and Open Problems

The authors explicitly note several limitations:

Single-Layer Theory: The $\pi/(\pi-2)$ constant is derived for a two-layer toy model; the behavior of the full transformer stack is not fully captured.
Gaussian Assumption: Real trained weights are not i.i.d. Gaussian, leading to deviations from the $2/\pi$ limit in empirical measurements.
Activation Functions: The closed-form results rely on ReLU's half-space structure. Extension to gated activations (SwiGLU/GeGLU) used in modern LLMs remains open, though experiments suggest qualitatively similar behavior.
Multi-Layer Propagation: The exact law governing how the radial fraction $R$ evolves across depth in residual streams remains an open problem.
Energy-to-Loss Mapping: A rigorous mapping from perturbation energy to Perplexity/NLL for non-linear regimes remains partially unresolved, though the paper provides strong empirical correlations.

In summary, the paper argues that the success of ternary quantization in pre-norm models is a geometric inevitability of the RMSNorm filter interacting with sign-preserving perturbations in a ReLU-activated space, while the extreme fragility of these models to sign flips is driven by heavy-tailed outlier features rather than architectural depth.