💬 NLP

Adaptive Block-Scaled Data Types

This paper introduces Adaptive Block-Scaled Data Types, specifically the IF4 format, which dynamically selects between FP4 and INT4 representations for each group of 16 values to overcome NVFP4's quantization error limitations, thereby achieving superior performance in language model quantization and demonstrating efficient hardware implementation.

Original authors: Jack Cook, Hyemin S. Lee, Kathryn Le, Junxian Guo, Giovanni Traverso, Anantha P. Chandrakasan, Song Han

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jack Cook, Hyemin S. Lee, Kathryn Le, Junxian Guo, Giovanni Traverso, Anantha P. Chandrakasan, Song Han

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Packing a Suitcase for a Trip

Imagine you are trying to fit a massive, complex world (a Large Language Model like the ones that power chatbots) into a tiny suitcase (computer memory).

To make the suitcase small enough to carry, you have to compress the clothes. In the world of AI, this "compression" is called quantization. Instead of using big, heavy 16-bit or 32-bit numbers, we try to squeeze everything into tiny 4-bit numbers.

The Problem:
Think of 4-bit numbers like a suitcase with only 16 specific slots.

The Old Way (NVFP4): Imagine you have a suitcase where the slots are arranged to hold mostly small socks, but occasionally a giant winter coat. If you try to stuff a winter coat into a "sock slot," it gets squished and ruined. This causes errors, especially with the "big" numbers in your data.
The Recent Fix (4/6): Some researchers tried to fix this by saying, "Okay, let's make the suitcase slightly smaller so the coat fits better." But this meant you had to throw away two other useful slots (values) to make it work. You were trading one problem for another.

The New Solution: The "Shape-Shifting" Suitcase (IF4)

The authors of this paper, led by Jack Cook and Song Han, invented a new type of suitcase called IF4 (Int/Float 4).

Here is how it works, using a simple analogy:

1. The "Group" Concept

Instead of looking at one number at a time, the AI looks at a group of 16 numbers together. Think of this group as a single "pocket" in your suitcase.

2. The Magic Switch

In the old suitcases, every pocket was rigid. It was either designed for socks (floating point) or for heavy boots (integers), but not both.

IF4 is a smart, shape-shifting pocket.
For every group of 16 numbers, the AI asks a simple question: "What do these numbers look like?"

Scenario A: If the numbers are mostly small with one giant outlier (like 15 socks and 1 coat), the pocket stays in Float Mode. It stretches to hold the coat.
Scenario B: If the numbers are all roughly the same size (like 16 pairs of shoes), the pocket instantly snaps into Integer Mode. This mode is more uniform and precise for evenly sized items.

3. The Secret Signal

How does the computer know which mode a pocket is in?
In the old system, there was a tiny "sign bit" (a flag) on the scale factor that was never used. The authors realized, "Hey, we have an empty flag! Let's use it!"

Flag = 0: "We are using Float Mode."
Flag = 1: "We are using Integer Mode."

This is like having a suitcase that automatically changes its internal dividers based on what you are packing, without needing any extra space in the bag.

Why is this a Big Deal?

1. Less "Squishing" (Lower Error)
Because the suitcase can choose the best shape for the specific group of numbers, it doesn't have to force a "coat" into a "sock" slot. This means the AI makes fewer mistakes when it reads the data. The paper shows that models trained with IF4 are smarter and make fewer errors than those using the old NVFP4 format.

2. No Extra Weight (No Memory Overhead)
Usually, when you add a "smart" feature to a suitcase, it gets heavier. But because IF4 uses a flag that was already there and unused, the suitcase weighs exactly the same as the old one. You get better packing without paying a price in storage space.

3. It Works on Future Hardware
The authors didn't just do math on a computer; they built a tiny prototype of the "machine" that would read these suitcases (a hardware unit called a MAC). They found that even though the machine has to do a tiny bit of extra thinking to check the flag, it's so fast that it doesn't slow anything down. It's like having a smart lock on your door that takes a split second longer to open but keeps your house much safer.

The "6/7" Trick

There is one technical detail that sounds like magic but is actually just clever math.

Float mode can hold a maximum value of 6.
Integer mode can hold a maximum value of 7.
To make them play nice together, the authors decided that if you use Integer mode, they will temporarily treat the numbers as if they are scaled by 6/7. This ensures that no matter which mode the pocket is in, the "biggest" number fits perfectly without overflowing the suitcase.

Summary

The paper introduces IF4, a new way to compress AI models.

Old Way: One size fits all (but fits poorly).
New Way (IF4): A smart, adaptive system that switches between two packing styles (Float vs. Integer) for every group of numbers.
Result: Smarter AI, fewer mistakes, and the same tiny suitcase size.

It's like upgrading from a rigid plastic box to a smart, self-adjusting suitcase that rearranges itself to fit your clothes perfectly, ensuring nothing gets crushed and everything fits.

1. Problem Statement

As Large Language Models (LLMs) grow in size, there is a critical need to reduce memory footprint and increase computational throughput through low-precision quantization (e.g., 4-bit). While formats like NVFP4 (NVIDIA's 4-bit floating point) have gained hardware support, they suffer from specific limitations:

Error Distribution: NVFP4 uses a block-scaled approach where groups of 16 values share a single FP8 scale factor. However, the fixed distribution of FP4 values leads to significant quantization errors, particularly for values near the maximum representable value in a group.
Limitations of Adaptive Scaling (4/6): A recent method called "Four Over Six" (4/6) attempted to mitigate this by adaptively scaling groups to a max value of 4 or 6. However, this approach has two major drawbacks:
1. Reduced Dynamic Range: Scaling to 4 requires reducing the global tensor scale factor, shrinking the dynamic range of the quantized tensor by ~43%.
2. Loss of Precision: Scaling to 4 forces the exclusion of the extreme FP4 values ( $\pm 6$ ), effectively discarding 2 out of the 15 useful representable values per group.
Training Challenges: Training with 4-bit numbers (W4A4G4) is difficult because the limited precision (only 16 values) causes instability, requiring complex overhead operations (e.g., Hadamard transforms, stochastic rounding) that negate the speed benefits of low precision.

2. Methodology: Adaptive Block-Scaled Data Types

The authors propose a new class of data types called Adaptive Block-Scaled Data Types, with the primary focus on IF4 (Int/Float 4).

Core Concept

Instead of forcing every group of 16 values into a single distribution (pure FP4 or pure INT4), IF4 allows each block to dynamically select the representation that minimizes quantization error for that specific block's data distribution.

Dual Representation: For each group of 16 values, the system evaluates two options:
1. Standard FP4: Scaled by an FP8 E4M3 scale factor (similar to NVFP4).
2. Scaled INT4: The values are converted to integers, scaled, and stored.
Selection Mechanism: The system calculates the Mean Squared Error (MSE) for both representations. It selects the format (FP4 or INT4) that yields the lower error.
Sign Bit Repurposing: In NVFP4, the sign bit of the FP8 scale factor is unused because individual values have their own sign bits. IF4 repurposes this sign bit as an indicator:
- 0: The group is stored as FP4.
- 1: The group is stored as scaled INT4.
The 6/7 Scaling Method: To ensure both formats cover the same dynamic range without reducing the global tensor scale:
- The maximum value of FP4 is 6.
- The maximum value of INT4 is 7.
- When a group selects INT4, the values are scaled by a factor of 7/6 before quantization and de-scaled by 6/7 during de-quantization (matrix multiplication). This ensures the effective range remains 0–6 for all groups, preserving the global dynamic range.

Extensions

The authors also propose IF3 (3-bit) and IF6 (6-bit) formats, applying the same logic of switching between floating-point and scaled integer representations based on input distribution.

3. Key Contributions

New Data Type (IF4): Introduction of a 4-bit format that adaptively switches between FP4 and INT4 per block, eliminating the need to sacrifice dynamic range or useful values (unlike the 4/6 method).
Zero Storage Overhead: The format uses the exact same bit-width and storage structure as NVFP4, utilizing the previously unused sign bit of the scale factor as a format indicator.
Hardware Feasibility: Design and synthesis of an IF4 Multiply-Accumulate (MAC) unit in 28nm CMOS, demonstrating that the format can be implemented in next-generation hardware with minimal latency and area overhead.
Comprehensive Evaluation: Extensive testing on pre-training (W4A4G4) and post-training quantization (PTQ) across multiple model sizes (up to 397B parameters) and tasks.

4. Results

Quantization Error

MSE Reduction: On normally distributed data ( $N(0,1)$ ), IF4 achieves a Mean Squared Error of 6.2 $\times 10^{-3}$ , significantly lower than NVFP4 (9.0), NVINT4 (7.4), and the 4/6 method (7.5).
Weight Channels: In Qwen3.5-35B, IF4 reduces quantization error across the majority of weight channels compared to existing block-scaled formats.

Pre-Training (W4A4G4)

Training Loss: IF4 achieves training loss closer to the high-precision BF16 baseline than NVFP4.
Hadamard Transform Synergy: The performance gap widens when using MS-EDEN (a technique involving Hadamard transforms in the backward pass). IF4 adapts better to the transformed inputs, selecting INT4 more frequently for these distributions, whereas NVFP4 struggles.

Post-Training Quantization (PTQ)

Perplexity: On WikiText-2 and C4 datasets, IF4 consistently outperforms NVFP4, NVINT4, and 4/6 across various model sizes (Nemotron 3 Nano, Qwen3.5).
- Example: For Qwen3.5-35B-A3B, IF4 achieves a WikiText-2 perplexity of 8.07, compared to 8.14 for NVFP4 and 8.31 for NVINT4.
Downstream Tasks: IF4 achieves the highest average accuracy across five tasks (ARC-Easy, ARC-Challenge, HellaSwag, LAMBADA, PIQA) for nearly all model configurations.

Hardware Performance

Latency: The IF4 MAC unit has a 4.7% higher latency (4.00ns vs 3.82ns) compared to a baseline NVFP4 MAC, primarily due to the extra floating-point multiplication required for range alignment.
Efficiency: While area and power increase by ~66% and ~28% respectively at the MAC unit level, the authors argue this is negligible at the system level. Modern AI accelerators are memory-bound; the small increase in compute latency does not significantly impact overall throughput dominated by memory bandwidth.

5. Significance

This work addresses a fundamental bottleneck in low-precision LLM training and inference: the rigidity of fixed numerical formats. By introducing Adaptive Block-Scaled Data Types, the authors demonstrate that:

Precision can be recovered without storage cost: Switching between formats per block allows the system to "choose the best tool for the job" without increasing memory usage.
Hardware compatibility is achievable: The proposed MAC unit proves that these adaptive formats can be implemented efficiently in silicon, paving the way for future hardware accelerators to natively support mixed-format quantization.
Training viability: The results suggest that W4A4G4 (fully 4-bit training) is more viable with IF4, potentially enabling faster training cycles and reduced memory requirements for frontier models.

In summary, IF4 represents a significant step forward in making 4-bit quantization a practical, high-performance standard for the next generation of LLMs.