🤖 AI

FOCUS: FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling

This paper introduces FOCUS, a post-training quantization framework that enhances FP4 accuracy for large language models by employing Coupled-Relaxation Scaling to decouple learnable quantization scales from hardware constraints and Dual-Granularity Scaling for finer weight adaptation, achieving state-of-the-art performance without additional inference overhead.

Original authors: Xianglong Yan, Hong Liu, Chengzhu Bao, Tianao Zhang, Guanghua Yu, Jianchen Zhu, Yulun Zhang

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xianglong Yan, Hong Liu, Chengzhu Bao, Tianao Zhang, Guanghua Yu, Jianchen Zhu, Yulun Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to fit a massive, intricate library of knowledge into a tiny, portable backpack. This is the daily struggle for "Large Language Models" (LLMs), the super-smart computer brains that can write stories, solve math problems, and chat like humans. The problem is that these brains are so huge they need enormous amounts of memory and power to run, making them expensive and slow to use on everyday devices. To fix this, scientists use a trick called "quantization." Think of this like translating a high-definition, 4K movie into a lower-resolution format so it fits on an old phone. The goal is to shrink the data without losing the plot. Recently, a new, super-tiny format called "FP4" has emerged, supported by modern computer chips, which promises to shrink these models even further. However, there's a catch: squeezing the data this small often makes the model "forget" things or get confused, losing accuracy. The big question is: how do we shrink the model to this tiny size without ruining its intelligence?

This paper introduces a clever new method called FOCUS to solve that exact problem. The researchers realized that the current way of shrinking these models was too rigid. Imagine you are packing a suitcase. The standard rule says: "You must use the exact same small, pre-made boxes to pack your clothes, and you must use those same boxes to unpack them later." The paper argues that this is silly. While you do need to use the small boxes to store the clothes (because that's what fits in the suitcase), you don't need to use those same tiny boxes to figure out how to fold the clothes in the first place. You could use a giant, flexible measuring tape to plan the packing, and then just fit the result into the small boxes.

FOCUS uses this insight to improve how models are prepared for the tiny FP4 format. It introduces two main tricks:

Coupled-Relaxation Scaling (CRS): This is the "flexible measuring tape." In the old way, the computer had to use a very rough, low-precision ruler to decide how to shrink the numbers, and it had to use that same rough ruler to put them back together. FOCUS says, "Let's use a super-precise, full-size ruler to do the shrinking math, and then squeeze the result into the rough box." This allows the computer to find a much better way to pack the data without breaking the rules of the tiny format.
Dual-Granularity Scaling (DGS): This is about packing smaller items. The old method treated a whole group of 32 numbers as one big chunk, using one ruler for all of them. But what if some numbers in that group are huge and others are tiny? FOCUS splits that big group into smaller sub-groups (like cutting a big pizza into slices) and gives each slice its own custom ruler. This lets the computer adapt to the specific details of the data, rather than forcing everything into a one-size-fits-all box.

The results are impressive. When the researchers tested FOCUS on several different AI models (like Qwen and LLaMA), it consistently outperformed all other methods. For example, on a model called Qwen3-4B, FOCUS managed to recover 98.2% of the original model's accuracy when using the NVFP4 format, and 96.2% with MXFP4. This is a significant jump compared to previous techniques, which often struggled to stay above 90% or 94%.

Crucially, the paper shows that FOCUS doesn't make the model slower or harder to use. Even though the computer does extra math while preparing the model (during the "training" or "calibration" phase), the final model runs just as fast as any other FP4 model. It takes about 19 minutes to prepare a Qwen3-4B model on a single high-end GPU, which is actually faster than some competing methods. The researchers emphasize that this is a "post-training" method, meaning they don't need to re-teach the model from scratch; they just tweak the packing instructions.

In short, FOCUS suggests that by loosening the rules on how we calculate the shrinkage (while keeping the final storage format strict), we can fit these giant AI brains into tiny spaces without losing their smarts. It's a bit like realizing you can fold a shirt perfectly using a large table, even if you only have a small drawer to store it in. The paper proves that this approach works better than the old, rigid ways, offering a path to run powerful AI on devices that currently can't handle them.

Technical Summary: FOCUS – FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling

Problem Statement

Large Language Models (LLMs) face significant deployment challenges due to their massive parameter counts, necessitating efficient compression techniques. While Post-Training Quantization (PTQ) has become a standard approach, pushing quantization to 4-bit floating-point (FP4) formats like MXFP4 and NVFP4 remains difficult. These formats offer native hardware support on modern accelerators (e.g., NVIDIA Blackwell) through block-wise scaling schemes (e.g., groups of 32 or 16 values sharing a low-precision scale).

However, maintaining model accuracy under FP4 precision is a persistent bottleneck. A critical limitation in existing methods is the tight coupling between the quantization scale ( $S_q$ ) and the dequantization scale ( $S_{dq}$ ). Current approaches force both scales to conform to the same discrete, low-precision hardware format (e.g., E8M0 for MXFP4) and the same block granularity. This is suboptimal because:

Redundant Constraints: Only the dequantization scale ( $S_{dq}$ ) is stored and used during inference, requiring it to adhere to hardware constraints. The quantization scale ( $S_q$ ), used solely during the offline quantization process to determine the best FP4 weight assignment, is discarded afterward and need not obey these constraints.
Discrete Optimization Space: By forcing $S_q$ into a discrete low-precision format, existing methods restrict the optimization landscape, preventing gradient-based learning from finding optimal weight assignments.
Granularity Mismatch: A single block-level scale cannot capture local weight variations within a block, leading to quantization errors.

Methodology

The authors propose FOCUS, a post-training quantization framework that leverages the "freedom" of the quantization scale to optimize FP4 accuracy without introducing inference overhead. The framework operates on two key dimensions: Precision and Granularity.

1. Baseline: End-to-End Scale Learning

The authors first establish a baseline where model weights are frozen, and block-wise scales are learned end-to-end using a knowledge distillation loss (KL-Top). Unlike traditional PTQ, this baseline optimizes scales via gradient descent using the Straight-Through Estimator (STE) to handle non-differentiable quantization operations. However, this baseline still enforces $S_q = S_{dq}$ , maintaining the tight coupling.

2. Coupled-Relaxation Scaling (CRS)

To address the precision constraint, FOCUS introduces Coupled-Relaxation Scaling (CRS).

Mechanism: CRS decouples the quantization scale from the hardware-mandated format by introducing a learnable full-precision coefficient, $c_i$ .
Formulation: The quantization scale is defined as $S_q = S_{dq} \cdot c_i$ , where $S_{dq}$ is the hardware-compliant scale (quantized to E4M3 or E8M0) and $c_i$ is a full-precision (FP16) learnable parameter initialized to 1.
Effect: This relaxes the tight coupling, lifting the optimization of $S_q$ from a discrete low-precision space to a continuous full-precision space. This allows the model to find better FP4 weight assignments while ensuring $S_{dq}$ remains compliant with hardware requirements. The coefficient $c_i$ is discarded after training.

3. Dual-Granularity Scaling (DGS)

To address the granularity constraint, FOCUS introduces Dual-Granularity Scaling (DGS).

Mechanism: DGS subdivides each hardware block (size 32 for MXFP4, 16 for NVFP4) into smaller sub-blocks (e.g., size 8). Each sub-block is assigned an independent full-precision relaxation coefficient ( $c_k$ ).
Formulation: The quantization for a sub-block $k$ becomes $\bar{W}_k = Q(W_k / (S_{dq} \cdot c_k))$ , while the dequantization scale $S_{dq}$ remains shared across the entire block.
Effect: This allows the quantization scale to operate at a finer granularity than the dequantization scale, capturing local weight variations within a block that a single block-level scale would miss. Like CRS, these coefficients are discarded post-training.

4. Deployment and Compatibility

FOCUS does not rely on structural transformations (e.g., Hadamard rotations) that often incur inference overhead or require framework modifications. The final deployed model consists strictly of standard FP4 weights and hardware-compliant dequantization scales, ensuring seamless compatibility with existing inference engines and zero additional inference cost.

Key Contributions

Insight Identification: The paper identifies that the quantization scale is free from hardware constraints, unlike the dequantization scale, and proposes exploiting this freedom in both precision and granularity.
Coupled-Relaxation Scaling (CRS): A technique that introduces a learnable full-precision coefficient to relax the coupling between quantization and dequantization scales, enabling continuous gradient-based optimization.
Dual-Granularity Scaling (DGS): A method that assigns quantization scales at the sub-block level, capturing intra-block weight variations that block-wise scaling cannot represent.
State-of-the-Art Performance: Extensive experiments demonstrate that FOCUS achieves SOTA FP4 accuracy under both MXFP4 and NVFP4 formats across multiple LLM families (Qwen, LLaMA) without inference overhead.

Experimental Results

The authors evaluated FOCUS on five LLMs (Qwen3-4B/8B, LLaMA-3.1-8B, LLaMA-3.2-1B/3B) using WikiText2, C4, and five zero-shot reasoning benchmarks (ARC, HellaSwag, PIQA, Winogrande).

Accuracy Recovery: On Qwen3-4B under NVFP4, FOCUS recovers 98.2% of the FP16 zero-shot accuracy, significantly outperforming baselines like RTN, GPTQ, MR-GPTQ, 4over6, and RaZeR.
Perplexity: FOCUS achieves the lowest perplexity in nearly all settings. For Qwen3-8B, it reaches 9.47 perplexity on WikiText2 (NVFP4), nearly matching the FP16 baseline of 9.73.
Reasoning Capabilities: On chain-of-thought tasks (GSM8K, MMLU), FOCUS consistently outperforms competing methods, preserving multi-step reasoning capabilities better than other FP4 quantization approaches.
Efficiency: The quantization process is lightweight. On a single NVIDIA H20 GPU, quantizing Qwen3-4B takes 19 minutes, which is 34% faster than the previous SOTA method (MR-GPTQ).

Significance and Claims

The paper claims that FOCUS represents a paradigm shift in FP4 quantization by rethinking the role of the quantization scale. By recognizing that the quantization scale does not need to be stored or adhere to hardware constraints, the authors unlock a significant, previously untapped optimization space.

The significance lies in achieving state-of-the-art accuracy for FP4 inference while maintaining zero inference overhead and seamless hardware compatibility. Unlike transformation-based methods that add computational cost or require framework adaptation, FOCUS operates purely within the scale optimization space. The authors assert that their approach effectively bridges the performance gap between FP4 and full-precision (FP16) models, making high-efficiency W4A4 inference more viable for practical deployment.