💬 NLP

Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling

This paper introduces "Four Over Six" (4/6), an adaptive block-scaling technique for NVFP4 quantization that reduces quantization error by dynamically adjusting block scales to better handle large values, thereby achieving training and inference performance closer to BF16 with minimal computational overhead.

Original authors: Jack Cook, Junxian Guo, Guangxuan Xiao, Yujun Lin, Song Han

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jack Cook, Junxian Guo, Guangxuan Xiao, Yujun Lin, Song Han

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to pack a massive library of books into a tiny, portable suitcase. The books represent the "brain" of a Large Language Model (AI), and the suitcase represents the computer's memory. To fit everything in, you need to shrink the books.

For a long time, we've been using a method called BF16 (a standard, high-precision format) which is like packing the books in sturdy, heavy boxes. It's accurate, but the suitcase gets full fast. Recently, engineers tried switching to NVFP4, a much smaller format. Think of NVFP4 as a set of tiny, lightweight origami boxes. They fit way more books in the suitcase and make the AI run faster, but there's a catch: because the boxes are so small, some of the "big" books get squished, crumpled, or lose their pages. This "crumpling" is called quantization error, and it makes the AI make mistakes.

The Problem: The "Rough Edge" of Small Boxes

The paper explains that NVFP4 has a weird quirk. It has specific sizes it can hold: 0.5, 1, 1.5, 2, 3, 4, and 6.

If a book is exactly size 4, it fits perfectly.
If a book is size 5, it doesn't fit. You have to force it into the size 4 box or the size 6 box.
If you force a size 5 book into a size 4 box, you lose a lot of information (it gets crushed). If you force it into a size 6 box, you waste a lot of space (it rattles around).

The authors found that the "crushing" happens most often with books that are almost as big as the biggest box (the "near-maximal" values). In the standard method, the suitcase is sized to hold the absolute biggest book possible (size 6). But this leaves a huge gap where books of size 5 get squished badly.

The Solution: "Four Over Six" (4/6)

The authors, Jack Cook and his team, came up with a clever trick called Four Over Six.

Imagine you are packing a suitcase. Instead of forcing every single item to fit into a "Size 6" box, you look at each group of items.

The Old Way: You assume every group needs a "Size 6" box. If a group has a book that is size 5, you crush it.
The 4/6 Way: You check the group. If the biggest book in that specific group is only size 5, you say, "Okay, we don't need a Size 6 box for this group. Let's use a Size 4 box instead."

By switching to a smaller "Size 4" box for that specific group, the "Size 5" book (which is now relatively smaller compared to the box limit) fits much more comfortably. It's no longer being crushed against the edge.

The Analogy:
Think of it like a video game character jumping.

Standard NVFP4: The game assumes the character can jump up to 10 feet. If they try to jump 9 feet, the game rounds it down to 8 feet (a big drop).
4/6 Method: The game checks the specific level. If the highest platform is only 6 feet high, it switches the "jump limit" to 6 feet. Now, a 5-foot jump is rounded to 6 feet (or 4 feet) much more accurately. The character doesn't fall as far.

How It Works in Practice

The paper describes a smart algorithm that looks at every small chunk of data (called a "block") before packing it:

It tries packing the chunk into a "Size 6" box and calculates how much information is lost.
It tries packing the same chunk into a "Size 4" box and calculates the loss.
It picks whichever box causes less damage (less error).

Usually, for chunks with very big numbers, the "Size 4" box is the winner because it makes the "near-maximal" numbers fit better.

The Results

The team tested this on a massive AI model called Nemotron 3 Nano (30 billion parameters).

Training: When they trained the AI using 4/6, the AI learned better and made fewer mistakes compared to the standard NVFP4 method. It got much closer to the performance of the heavy, slow "BF16" method, but kept the speed and size benefits of NVFP4.
Speed: They showed that this "smart packing" doesn't slow down the computer. It adds less than 15% extra work, which is a tiny price to pay for much better accuracy.
Existing Models: They also tried this on already-trained models (like Llama and Qwen). Even without re-training, using 4/6 made these models smarter and more accurate on tests like reading comprehension and logic puzzles.

The Bottom Line

The paper claims that by simply being smarter about which box size to use for different groups of data—sometimes using a "4" instead of a "6"—you can stop the "crushing" of important information. This makes low-precision AI (NVFP4) much more accurate, allowing us to run bigger, faster AI models on current hardware without them losing their "brainpower."

They even released the code (kernels) so others can use this "smart packing" method on NVIDIA's newest Blackwell GPUs.

Technical Summary: Four Over Six (4/6)

Problem Statement

As Large Language Models (LLMs) scale, there is increasing interest in low-precision numerical formats like NVFP4 to improve training speed and reduce memory usage. However, quantizing models to NVFP4 remains challenging due to precision loss, which degrades model performance.

The core issue identified in this work is the non-uniform step size inherent in floating-point formats. Unlike integer formats, FP4 (specifically E2M1) has dynamic step sizes: 0.5 between 0 and 2, 1 between 2 and 4, and 2 between 4 and 6. In standard NVFP4 block-scaled quantization, values are scaled to the full range of [-6, 6]. This creates a significant quantization gap for near-maximal values. Specifically, when a block's maximum value is scaled to 6, the representable FP4 value "4" corresponds to only 66.6% of the block's maximum magnitude. Consequently, any values between 66.6% and 100% of the block's maximum cannot be represented accurately, leading to large quantization errors for these near-maximal values.

The authors' analysis of Llama-3.1-8B indicates that while scale factor quantization errors have minimal impact, the error introduced by rounding near-maximal values (specifically around a scaled value of 5) is the primary driver of performance degradation in NVFP4 quantized models.

Methodology: Four Over Six (4/6)

The paper introduces Four Over Six (4/6), a modification to the NVFP4 block-scaled quantization algorithm designed to reduce quantization error for near-maximal values.

Core Mechanism

Instead of uniformly scaling all blocks to the maximum FP4 value of 6, 4/6 adaptively scales specific blocks to a maximum of 4.

Standard NVFP4 (Scale = 6): Values are spread across [-6, 6]. The gap between representable values 4 and 6 is 2 units.
4/6 (Scale = 4): For selected blocks, values are spread across [-4, 4]. The gap between representable values 3 and 4 is 1 unit.

By sacrificing the ability to represent the extreme values of 5 and 6 in specific blocks, the method allows for a more uniform distribution of representable values in the 75%–100% range of the block's magnitude. For example, a value representing 75% of a block's maximum can be represented exactly as "3" in a scale-4 block, whereas it would fall into a large error gap in a scale-6 block.

Adaptive Block Scaling Algorithm

Since scaling all blocks to 4 is not universally beneficial (as it reduces the dynamic range of the block by 50% compared to scaling to 6), the method employs an adaptive selection process:

Dual Quantization: For each block, the algorithm quantizes the values twice: once with a scale factor of 6 and once with a scale factor of 4.
Error Calculation: The quantization error for both versions is calculated against the original high-precision values.
Selection Rule: The algorithm selects the scale (4 or 6) that minimizes the error. The paper evaluates three selection rules:
- Minimum Maximum Error
- Minimum Mean Absolute Error (MAE)
- Minimum Mean Squared Error (MSE)
- Finding: For Post-Training Quantization (PTQ), MSE yields the best results. For Pre-Training, MAE performs slightly better.

Implementation Details

Tensor Scale Adjustment: To prevent overflow in the FP8 E4M3 scale factors when a block is scaled to 4 (which requires a scale factor 1.5x larger than a block scaled to 6), the global tensor scale $\alpha$ is computed using a maximum FP8 value of 256 instead of the standard 448.
Hardware Efficiency: The method is implemented using NVIDIA Blackwell GPU PTX instructions (cvt family) within a CUDA kernel. All intermediate values and error calculations are kept in the register file to minimize memory access overhead.
Overhead: The authors report that 4/6 adds less than 15% computational overhead to the NVFP4 quantization kernel.

Key Contributions and Results

1. Pre-Training Performance

The authors evaluated 4/6 on the Nemotron 3 Nano 30B-A3B model architecture using a 1-trillion-token dataset.

Loss Reduction: 4/6 brought the training loss 13.0% closer to the BF16 baseline compared to the standard NVFP4 training recipe.
Optimal Configuration: Surprisingly, applying 4/6 only to activations yielded performance comparable to applying it to activations, weights, and gradients, while introducing less computational overhead.
Scale Selection: In pre-training, using MAE for scale selection outperformed MSE.

2. Post-Training Quantization (PTQ)

The method was tested on Llama 3 and Qwen 3 models (ranging from 1B to 70B parameters) across WikiText-2 and C4 perplexity metrics, combined with existing PTQ methods (RTN, GPTQ, AWQ, SmoothQuant).

General Improvement: 4/6 uniformly improved perplexity metrics when combined with AWQ and SmoothQuant, bringing word perplexity 19.9% and 5.3% closer to BF16 performance, respectively.
GPTQ Interaction: 4/6 slightly degraded performance when combined with GPTQ in the current implementation, though the authors note that modifying the GPTQ optimization process to incorporate 4/6 could yield future improvements.
Downstream Tasks: On tasks like BoolQ, ARC-Easy, ARC-Challenge, and HellaSwag, 4/6 improved average task performance in nearly all cases when combined with AWQ and SmoothQuant.

3. Efficiency

The implementation is designed for modern hardware accelerators (NVIDIA Blackwell), ensuring that the benefits of reduced quantization error are not negated by excessive computational overhead.

Significance and Claims

The paper claims that 4/6 is a lightweight, efficient modification to the NVFP4 quantization algorithm that addresses the specific weakness of floating-point formats regarding near-maximal values.

Accuracy vs. Overhead: It achieves significant accuracy gains (closer to BF16 baselines) with minimal computational overhead (<15%), making NVFP4 training more viable without resorting to expensive "healing" phases or high-precision fallbacks.
Hardware Compatibility: The method is natively compatible with current NVIDIA Blackwell GPUs, leveraging existing hardware support for NVFP4.
General Applicability: While focused on NVFP4, the concept of adaptive block scaling is presented as a general technique to improve the representation of near-maximal values in block-scaled floating-point formats, though the authors note it is not directly applicable to MXFP4 due to differences in scale factor precision (FP8 E8M0 vs. E4M3).

The authors conclude that 4/6 represents a step forward in making low-precision FP4 training and inference practical for large-scale LLMs, bringing performance closer to high-precision baselines while maintaining the speed and memory benefits of quantization.