Pretraining Large Language Models with NVFP4

This paper introduces a novel NVFP4 training framework that combines Random Hadamard transforms, 2D quantization, stochastic rounding, and selective high-precision layers to successfully pretrain a 12-billion-parameter model on 10 trillion tokens with performance comparable to FP8 baselines, thereby demonstrating the viability of stable 4-bit precision training for large language models.

The Gist

Imagine you are trying to teach a super-smart robot (a Large Language Model, or LLM) to understand the entire internet, write poetry, solve math problems, and code software. To do this, you need to show it trillions of examples.

In the past, teaching this robot required a massive, energy-hungry "brain" that used 8-bit precision (like a high-definition camera). It was powerful, but it was expensive, slow, and ate up a lot of electricity.

NVIDIA's new paper introduces a way to teach this robot using 4-bit precision (NVFP4). Think of this as switching from a 4K camera to a highly optimized, ultra-efficient 1080p camera. You might think, "Won't the picture look blurry? Won't the robot make mistakes?"

The answer is: Not if you use the right tricks.

Here is the simple breakdown of how they made this work, using some everyday analogies.

1. The Problem: The "Tiny Box" Issue

Imagine you are trying to pack a giant, heavy suitcase (the data) into a tiny backpack (the 4-bit format).

• The Old Way (MXFP4): You try to force everything in. If the suitcase has a few really heavy items (outliers), they crush the rest of the clothes, or the backpack rips. The robot gets confused because the "heavy" numbers get squished into zero or the maximum limit.
• The New Way (NVFP4): NVIDIA designed a smarter backpack. Instead of one big compartment, they broke the backpack into smaller, specialized pockets.
  • Smaller Pockets: They use smaller blocks of data (16 items instead of 32). This means the "heavy" items don't crush as many "light" items.
  • Better Labels: They use a more precise label system (E4M3) to describe how heavy the items are, rather than just guessing with powers of two. This ensures the backpack fits the data perfectly without tearing.

2. The Four Magic Tricks

Just having a better backpack isn't enough. To make the robot learn as well as it did with the big 8-bit camera, they added four specific "training techniques":

A. The "Safety Net" (Mixed Precision)

Imagine you are teaching a student to do complex math. You let them do 90% of the work with a cheap calculator (4-bit), but for the final, most critical steps (like the last few pages of a thesis), you let them use a super-precise, expensive calculator (BF16/FP32).

• Why? The beginning and middle of the learning process are robust, but the very end is delicate. If you use the cheap calculator for the final steps, the student might get the answer slightly wrong. By keeping the last few layers in high precision, the robot gets the best of both worlds: speed and accuracy.

B. The "Shuffle" (Random Hadamard Transforms)

Imagine you have a deck of cards where a few cards are "Jokers" (outliers) that are huge and weird. If you deal them out, they mess up the game.

• The Trick: Before dealing the cards, you shuffle the deck using a special mathematical "shuffle" (Hadamard transform). This spreads the "Jokers" out so they aren't clumped together. Now, when the robot looks at the data, the weird numbers look like normal numbers, and the 4-bit format can handle them easily.
• Note: They found this shuffle is only needed for the "backward pass" (when the robot checks its mistakes), not the "forward pass" (when it makes predictions).

C. The "Consistent Map" (2D Scaling)

Imagine you are giving directions to a friend.

• The Problem: In the morning, you tell them to turn "Left" based on the street name. In the evening, when they retrace their steps, you tell them to turn "Right" based on the building number. If the map changes between morning and night, your friend gets lost.
• The Fix: In AI, the "forward pass" and "backward pass" look at data differently (rows vs. columns). If the 4-bit format treats them differently, the robot gets confused. NVIDIA created a 2D block scaling method. It's like drawing a grid on the map so that no matter which way you look at it, the "Left" and "Right" instructions remain consistent. This keeps the robot from getting lost during learning.

D. The "Fair Coin Flip" (Stochastic Rounding)

Imagine you have a jar of marbles, and you need to round the number of marbles to the nearest whole number.

• The Problem: If you always round 1.4 down to 1, and 1.6 up to 2, you introduce a bias. Over millions of calculations, the robot drifts in one direction.
• The Fix: Instead of always rounding down, you flip a coin. If you have 1.4, you round down 60% of the time and up 40% of the time. This "randomness" ensures that, on average, the robot stays perfectly centered and doesn't drift off course. This is crucial for the "gradient" (the error signal) that tells the robot how to learn.

3. The Results: Did it Work?

NVIDIA tested this on a massive 12-billion-parameter model, training it on 10 trillion tokens (a huge amount of data).

• The Comparison: They compared their 4-bit robot to a standard 8-bit robot.
• The Outcome: The 4-bit robot performed almost identically to the 8-bit one.
  • On a test called MMLU-Pro (a hard exam for AI), the 8-bit robot got 62.62%. The 4-bit robot got 62.58%.
  • The loss curve (a measure of how well the robot is learning) was almost a perfect match.

The Bottom Line

NVIDIA has figured out how to shrink the "brain" of a super-intelligent AI by half (from 8-bit to 4-bit) without making it dumber.

• Why does this matter? It means we can train the next generation of AI models twice as fast, use half the memory, and save a massive amount of energy.
• The Analogy: It's like discovering a way to fly a jumbo jet using half the fuel, without losing any speed or safety.

This isn't just a small tweak; it's a major leap forward that makes the future of AI faster, cheaper, and more accessible.

Technical Summary

Here is a detailed technical summary of the paper "Pretraining Large Language Models with NVFP4" by NVIDIA.

1. Problem Statement

Training frontier Large Language Models (LLMs) requires massive computational resources (tens to hundreds of yottaflops), time, and energy. While 8-bit floating point (FP8) training has become standard to improve efficiency, transitioning to even narrower 4-bit floating point (FP4) precision offers the potential for 2–3x arithmetic speedups and 50% memory reduction on next-generation hardware (NVIDIA Blackwell).

However, training LLMs at 4-bit precision faces significant hurdles:

• Training Instability: Extreme quantization leads to convergence issues, particularly over long token horizons (trillions of tokens).
• Outlier Sensitivity: LLMs contain "outliers" (values with large magnitudes) that, when quantized to 4-bit, cause saturation or loss of dynamic range, degrading model accuracy.
• Chain Rule Violations: Standard quantization methods often result in inconsistent representations between the forward and backward passes, breaking the mathematical chain rule required for gradient descent.
• Quantization Bias: Deterministic rounding introduces systematic errors in gradient estimation, hindering convergence.

2. Methodology

The authors propose a comprehensive training methodology centered around NVFP4, a proprietary 4-bit format, combined with four key algorithmic techniques to ensure stability and accuracy.

A. The NVFP4 Format

NVFP4 improves upon the standard OCP MXFP4 format through specific design choices:

• Smaller Block Size: Reduces block size from 32 elements (MXFP4) to 16 elements. This narrows the dynamic range within each block, allowing values to fit better into the limited 4-bit range.
• Precise Scaling: Uses an E4M3 format for block scale factors (instead of UE8M0 in MXFP4), trading exponent range for additional mantissa bits to represent scales more precisely.
• Two-Level Scaling: Employs a global FP32 tensor-level scale factor combined with the local E4M3 block scale. This ensures that the maximum values in a block are mapped closer to the FP4 maximum representable value, minimizing wasted dynamic range and saturation.

B. Key Training Techniques

To enable stable 4-bit pretraining, the authors integrate the following components:

1. Mixed Precision (Selective High-Precision Layers):

   • Not all layers are equally sensitive to 4-bit quantization. The authors retain the first two blocks and the final eight blocks (approx. 15–16% of linear layers) in BF16.
   • The final layers are critical because they require higher dynamic range and mantissa precision to prevent training divergence.

2. Random Hadamard Transforms (RHT):

   • Applied specifically to weight gradient (Wgrad) inputs.
   • RHT redistributes outliers into an approximately Gaussian distribution, making them easier to represent in narrow formats.
   • The authors use a 16×16 Hadamard matrix size and a single fixed random sign vector shared across all layers, which was found to be optimal for large-scale models.

3. Two-Dimensional (2D) Block Scaling:

   • Problem: In standard 1D scaling, the dot-product dimension changes between forward (rows) and backward (columns) passes due to transposition, leading to different quantized representations for the same weights. This violates the chain rule.
   • Solution: Weights are grouped and scaled in 16×16 blocks (16 input channels × 16 output channels). This ensures the quantized representation of weights remains consistent across both forward and backward passes, preserving the chain rule.
   • Activations and gradients use standard 1×16 scaling, as they are less sensitive to this inconsistency.

4. Stochastic Rounding (SR):

   • Applied to gradient tensors (both Dgrad and Wgrad).
   • Replaces deterministic rounding (e.g., round-to-nearest-even) with probabilistic rounding. This prevents systematic bias in gradient updates, which is crucial for convergence in 4-bit training.
   • SR is not applied to forward pass tensors or weights, as it can amplify quantization error in those contexts.

3. Key Contributions

• First Large-Scale 4-bit Pretraining: Successfully trained a 12-billion parameter hybrid Mamba-Transformer model on 10 trillion tokens using 4-bit precision. This is the longest publicly documented training run in 4-bit precision to date.
• NVFP4 Format Validation: Demonstrated that NVFP4, with its smaller block size and precise scaling, outperforms the standard MXFP4 format in training stability and final model quality.
• Comprehensive Methodology: Established a robust recipe for 4-bit training involving mixed precision, 2D scaling, RHT, and stochastic rounding, validated through extensive ablation studies.
• Hardware Integration: Confirmed that these algorithms are fully supported and optimized on NVIDIA Blackwell GPUs (GB200/GB300) via the Transformer Engine.

4. Results

The study compared the NVFP4-trained model against an FP8 baseline:

• Training Loss: The NVFP4 model's validation loss closely tracked the FP8 baseline. During the stable training phase, the relative loss error remained below 1%, widening only slightly to ~1.5% during the learning rate decay phase.
• Downstream Accuracy: The 4-bit model achieved accuracy nearly identical to the FP8 baseline across diverse benchmarks:
  • MMLU-Pro: 62.58% (NVFP4) vs. 62.62% (FP8).
  • MMLU: 76.57% vs. 77.36%.
  • Math (MATH): 81.48% vs. 83.32%.
  • General Reasoning: 69.82% vs. 68.99%.
  • Note: The coding task (MBPP+) showed a slight drop (55.91% vs. 59.11%), which the authors attribute to evaluation noise rather than a fundamental failure of the method.
• NVFP4 vs. MXFP4: In side-by-side experiments on an 8B model, NVFP4 converged to a lower loss than MXFP4. To match NVFP4's performance, MXFP4 required 36% more training tokens (1.36T vs. 1T), highlighting NVFP4's superior efficiency.
• Ablation Studies: Removing any single component (RHT, 2D scaling, SR, or mixed precision) resulted in training divergence or significantly worse convergence, proving that all components are essential for large-scale 4-bit training.

5. Significance

This paper represents a major milestone in efficient LLM training:

1. Scalability: It proves that 4-bit precision is viable for training frontier models (10B+ parameters) on multi-trillion token datasets, a regime previously thought to require higher precision (FP8 or BF16).
2. Efficiency: By enabling 4-bit training, organizations can significantly reduce the memory footprint and computational cost of pretraining, potentially lowering the barrier to entry for training larger models or reducing the carbon footprint of AI development.
3. Algorithmic Innovation: The combination of 2D scaling and stochastic rounding addresses fundamental mathematical challenges (chain rule violations and bias) inherent in low-precision training, providing a blueprint for future narrow-precision algorithms.
4. Hardware Synergy: The work demonstrates the full potential of NVIDIA Blackwell's FP4 Tensor Cores, bridging the gap between hardware capabilities and software algorithms.

In conclusion, NVIDIA's NVFP4 approach, supported by a tailored training methodology, successfully unlocks the efficiency of 4-bit pretraining without sacrificing the performance of state-of-the-art language models.