🤖 machine learning

Normalized Architectures are Natively 4-Bit

This paper demonstrates that nGPT, an architecture constraining weights and hidden states to a unit hypersphere, enables stable and efficient end-to-end 4-bit training by naturally enhancing signal-to-noise ratios through constructive signal accumulation, thereby eliminating the need for complex precision-preserving interventions.

Original authors: Maxim Fishman, Brian Chmiel, Ron Banner, Daniel Soudry, Boris Ginsburg

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Maxim Fishman, Brian Chmiel, Ron Banner, Daniel Soudry, Boris Ginsburg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Cracking the Code with Low Precision

Imagine you are trying to build a massive, incredibly complex Lego castle (a Large Language Model). Usually, you use giant, sturdy bricks (high-precision math) to make sure the castle stands tall and doesn't crumble.

However, to make the castle faster to build and cheaper to store, engineers want to use tiny, fragile 4-bit Lego bricks. The problem is that when you use these tiny bricks with standard designs, the castle often collapses or becomes wobbly. To fix this, builders usually have to add expensive scaffolding, extra glue, and complex measuring tools (like "Randomized Hadamard Transforms" and "per-tensor scaling") just to keep the structure from falling apart. These fixes slow everything down.

The Solution: A New Blueprint (nGPT)

This paper introduces a new blueprint called nGPT. Instead of trying to patch up the fragile 4-bit bricks with extra glue, the authors changed the shape of the bricks themselves.

In nGPT, every piece of the model is forced to stay on a "unit hypersphere."

The Analogy: Imagine a standard Lego brick can be any size, from a tiny pebble to a giant boulder. This makes it hard to stack them neatly if they are all tiny.
The nGPT Twist: In nGPT, every brick is forced to be exactly the same size and shape, like a perfect marble. They are all constrained to sit on the surface of an invisible sphere.

The paper claims that because of this shape constraint, the model is natively robust. It can be built entirely out of the tiny 4-bit bricks without needing any of the extra scaffolding or glue. It just works.

Why Does It Work? The "Chorus" Effect

The researchers asked: Why does this sphere shape make the model so strong?

They looked at how the model does math. It does this by taking thousands of numbers, multiplying them, and adding them all up (a "dot product").

The Standard Way (GPT): Imagine a choir of 4,000 singers. In a standard model, the singers are all singing different notes at different volumes. When you add them up, the noise (mistakes made by the tiny 4-bit bricks) cancels out, but the actual song (the signal) gets lost in the chaos. The signal is weak.
The nGPT Way: Because all the singers (numbers) are forced to be the same size (on the sphere), they naturally start to sing in a slightly coordinated way. They aren't perfectly in sync, but they have a weak, positive correlation.
- The Analogy: It's like a crowd doing "The Wave" in a stadium. Even if the wave is small, because everyone is moving in the same general direction, the wave builds up into a huge, powerful motion.
- The Result: In nGPT, the "signal" (the intended math) builds up constructively, like a coordinated wave. The "noise" (the errors from the tiny bricks) still cancels itself out like random chatter.

This creates a much clearer signal. The paper calls this a higher Signal-to-Noise Ratio (SNR).

The "Flat Landscape" Benefit

Because the signal is so strong and clear, the model becomes very stable.

The Analogy: Imagine walking on a mountain.
- Standard Model: It's like walking on a jagged, rocky cliff. If you take a slightly wrong step (a bad learning rate or a tiny math error), you might fall off a ledge. You have to be very careful.
- nGPT Model: It's like walking on a wide, flat plateau. You can take big steps or small steps, and you won't fall off. You can walk in any direction without worrying about crashing.

This means engineers don't have to spend hours tweaking the "learning rate" (how fast the model learns). The settings that work for the big, heavy bricks also work perfectly for the tiny 4-bit bricks.

What They Tested

The authors didn't just talk about theory; they built and tested these models:

Small Scale: They tested a 1.2 billion parameter model.
Large Scale: They tested massive "Mixture of Experts" models with up to 30 billion parameters.
The Result: In every case, the nGPT models trained successfully using only 4-bit math. They didn't need the extra "glue" (RHT or scaling) that standard models require. In fact, the nGPT models were often more accurate and faster because they didn't have to do the extra work to fix the math errors.

The Bottom Line

The paper argues that we shouldn't just try to fix low-precision math with complicated patches. Instead, we should design the architecture itself to be "quantization-ready." By forcing the model into a spherical shape, the math naturally becomes robust enough to handle the tiny 4-bit bricks, making AI faster, cheaper, and easier to train without losing quality.

Technical Summary: Normalized Architectures are Natively 4-Bit

Problem Statement

Deploying large language models (LLMs) at 4-bit precision is essential for efficiency, yet training these models remains fragile. Standard transformer architectures typically require complex, overhead-heavy interventions to maintain stability during low-bit training. These interventions include Randomized Hadamard Transforms (RHT) to disperse outliers, dynamic per-tensor scaling, and mixed-precision exceptions. While these methods enable 4-bit training, they raise a fundamental question: Can an architecture possess intrinsic robustness to low-precision arithmetic without relying on such "quantization tricks"?

Methodology

The authors investigate nGPT, a transformer architecture that constrains both model weights and hidden representations to the unit hypersphere. The study focuses on NVIDIA's NVFP4 format, natively supported by Blackwell GPUs.

The core methodology involves a structural analysis of a 3.6B parameter transformer (based on the nanoGPT baseline) to trace the origins of quantization robustness. The authors decompose the matrix multiplication operation ($Y = WX$) into signal and noise components under NVFP4 quantization. They specifically analyze:

Signal-to-Noise Ratio (SNR): Measuring SNR at individual element levels versus the final dot-product summation.
Correlation Structure: Quantifying the pairwise correlations between element-wise products ( $w_k x_k$ ) within the dot product sum.
Scaling Laws: Examining how the SNR advantage scales with hidden dimension size ( $D$ ).
Empirical Validation: Training diverse architectures—including a 1.2B dense model and Hybrid Mamba-Transformer Mixture-of-Experts (MoE) models ranging from 400M to 30B parameters—using NVFP4 without RHT or dynamic scaling.

Key Findings and Mechanisms

1. Robustness Stems from Signal Accumulation, Not Noise Suppression

Contrary to the prevailing logic that low-bit robustness requires reducing quantization noise, the authors find that nGPT does not make individual weights or activations easier to quantize. The SNR of individual elements and their element-wise products is nearly identical between standard GPT and nGPT (~19 dB and ~16.5 dB, respectively).

The divergence occurs only at the summation step (the dot product).

Standard GPT: The signal accumulates incoherently, and noise averages out slowly, resulting in a final dot-product SNR of ~18.6 dB.
nGPT: The hypersphere constraint induces a constructive accumulation of the signal. The final dot-product SNR reaches ~26 dB, a ~7.4 dB advantage.

2. The Role of Positive Correlations

The mechanism driving this advantage is the coherence of signal accumulation.

In standard GPT, the network can rely on a few large, unbounded coordinates to dominate the dot product.
In nGPT, the normalization constraint blocks this path. To produce a large output, the model must learn distributed alignments across thousands of dimensions.
This dynamic induces weak but systematic positive correlations among the element-wise products ( $w_k x_k$ ). While noise correlations remain near zero in both architectures, signal correlations in nGPT are approximately 14 times larger than in GPT ( $\bar{\rho}_s \approx 1.32 \times 10^{-3}$ vs. $9.31 \times 10^{-5}$ ).
These positive correlations cause the true signal to grow faster than the incoherent noise during summation, effectively increasing the SNR.

3. Scaling with Width

The SNR advantage is not static; it compounds with model width. Theoretical analysis suggests three regimes based on hidden dimension $D$ :

Small Width: No significant gap.
Intermediate Width (Current Models, e.g., $D=4096$ ): The SNR ratio grows linearly with $D$ .
Large Width: The ratio saturates at a theoretical maximum of ~14.2.
The authors note that current models operate in the growth regime, implying that nGPT's advantages will become even more pronounced as models scale to larger hidden dimensions (e.g., $D=16384$ ).

4. Flatter Loss Landscape and Hyperparameter Robustness

The improved per-layer SNR translates to a flatter loss landscape. When weights are perturbed (simulating quantization noise), the validation loss of nGPT degrades 3.5 times slower than standard GPT.

Learning Rate Insensitivity: nGPT maintains a flat optimum across a wide range of learning rates in both BF16 and NVFP4. The optimal learning rate for BF16 transfers directly to NVFP4 without retuning. In contrast, standard GPT requires a separate, significantly higher learning rate sweep for NVFP4.

Experimental Results

The authors validated these findings across multiple configurations:

1.2B Dense Model: Trained on 1T tokens. nGPT in NVFP4 (without RHT or scaling) achieved lower relative error than standard GPT in NVFP4 and outperformed standard GPT in BF16 across downstream tasks (e.g., MMLU, GSM8K).
Hybrid Mamba-Transformer MoE: Evaluated on 400M/600M and 3B/30B parameter models. nGPT maintained stability and achieved near 0% relative error in the 3B/30B configuration without standard interventions.
Throughput: By removing the computational overhead of RHT and dynamic per-tensor scaling, nGPT NVFP4 achieves a 3.3–3.6× speedup over BF16 GPT baselines on Blackwell GPUs at large hidden sizes.

Significance and Claims

The paper claims a shift in the paradigm of low-precision model design. Rather than treating quantization as a compression problem to be patched with post-hoc algorithms, the authors argue that quantization robustness can be architectural.

The primary significance lies in demonstrating that the hyperspherical geometry of nGPT inherently biases the model toward a representation where signal survives 4-bit arithmetic unusually well. This structural property eliminates the need for overhead-heavy correction mechanisms (RHT, dynamic scaling), enabling stable, end-to-end NVFP4 training that is both more efficient and more robust than standard approaches. The authors suggest that as the industry moves toward 4-bit precision and below, such structural properties may prove more valuable than increasingly elaborate correction mechanisms layered on top of standard architectures.