🤖 AI

ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs

ARCQuant is a novel framework that achieves state-of-the-art accuracy and significant inference speedups for Large Language Models by boosting NVFP4 quantization through augmented residual channels, which effectively compensates for quantization errors while maintaining hardware-unified precision and utilizing standard GEMM kernels.

Original authors: Haoqian Meng, Yilun Luo, Yafei Zhao, Wenyuan Liu, Peng Zhang, Xindian Ma

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haoqian Meng, Yilun Luo, Yafei Zhao, Wenyuan Liu, Peng Zhang, Xindian Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly detailed library of knowledge (a Large Language Model, or LLM). To make this library fit into a small backpack (your computer's memory) and run quickly, you need to shrink the books. This process is called quantization.

Usually, you can shrink the books by summarizing them into 8-bit "paperbacks." But to get even smaller and faster, researchers are trying to shrink them into tiny 4-bit "postcards."

However, there's a problem. Some pages in these libraries have huge, dramatic paragraphs (called outliers) that don't fit well on a tiny postcard. If you try to squash them down, the whole page gets distorted, and the story makes no sense.

The Problem with Current Solutions

Scientists have tried two main ways to fix this, but both have flaws:

The "Shuffle" Method: Imagine trying to fix a messy room by shuffling all the furniture around so the big sofa fits in a small corner. While the sofa fits, you've now made the whole room chaotic. In AI terms, this "rotation" spreads the big numbers around, ruining the neat, organized blocks that the new computer chips (NVIDIA's Blackwell architecture) rely on to work fast.
The "Mixed-Size" Method: Imagine keeping the big sofa in a giant box (high precision) and the small chairs in tiny boxes (low precision). The problem is, your delivery truck (the computer hardware) is designed to carry only one size of box at a time. Mixing sizes forces the truck to stop and switch gears, slowing everything down.

The Solution: ARCQuant (The "Residual Backpack")

The authors of this paper, from Tianjin University, propose a new method called ARCQuant. Instead of shuffling the furniture or using mixed boxes, they use a clever trick: The Augmented Residual Channel.

Here is the analogy:

Imagine you are packing a suitcase (the data) for a trip.

The Main Item: You have a large, bulky winter coat (the "outlier" data).
The Problem: The suitcase is too small to fit the coat flat.
The Old Way: You try to force it in, crushing the coat (losing accuracy), or you buy a second, different-sized suitcase (mixed precision), which the airline won't let you check in efficiently.
The ARCQuant Way: You take the coat, fold it up tightly, and put it in a special, thin "residual" pocket attached to the side of the suitcase.
- The main suitcase holds the rest of your clothes perfectly.
- The thin pocket holds the specific details of the coat that got squished.
- Crucially: The airline (the computer hardware) sees this as one single, unified suitcase. They don't have to stop and switch gears. They just process the whole thing as one unit.

How It Works in Simple Terms

Identify the Trouble: The system scans the data to find the "bulky coats" (the outlier numbers that are too big for 4-bit).
Separate and Save: It takes those big numbers, calculates what would be lost if they were squished, and saves that "lost difference" (the residual) in a special extra column.
Unified Packing: It packs the main data and the "difference" data together into a single, standard format that the computer chip understands perfectly.
The Magic Math: When the computer reads the suitcase, it adds the main part and the "difference" part back together instantly. Because it's all done in one go, it's incredibly fast.

What the Paper Claims

The researchers tested this on popular AI models (like LLaMA and Qwen) using the newest NVIDIA graphics cards (RTX 5090 and PRO 6000).

Accuracy: Their method is so good that the AI performs almost exactly as well as the full-size, uncompressed version. It beats all other 4-bit methods currently available.
Speed: Because it doesn't force the computer to switch between different data formats, it runs up to 3 times faster than the standard high-precision version.
Efficiency: It uses less memory and doesn't slow down the computer, even though it's doing extra math to fix the "bulky coats."

In a Nutshell

ARCQuant is like a smart packing system for AI. It finds the items that are too big for a tiny box, wraps them in a special thin layer, and attaches them to the main box. This way, the AI stays smart and accurate, but fits into a smaller, faster package that the computer's hardware can handle without any hiccups.

Technical Summary: ARCQuant

Problem Statement
The deployment of Large Language Models (LLMs) is increasingly constrained by memory bandwidth and computational latency. While Post-Training Quantization (PTQ) is a standard optimization, achieving aggressive 4-bit weight and 4-bit activation (W4A4) quantization while maintaining model fidelity remains a formidable challenge. Recent hardware advancements, specifically the NVIDIA Blackwell architecture, introduce native support for fine-grained Microscaling formats like NVFP4. However, existing PTQ strategies are ill-suited for these formats:

Rotation-based methods (e.g., Hadamard transforms) redistribute outlier magnitudes across dimensions, disrupting the fine-grained block isolation inherent to NVFP4 and inflating local dynamic ranges.
Mixed-precision approaches (e.g., Atom, MicroMix) attempt to preserve sensitive channels in higher precision (INT8/FP16). However, these violate the hardware requirement for unified-precision computation. NVFP4 requires a block size of 16, while higher-precision formats often use 32, creating structural incompatibility that prevents the use of optimized Tensor Core instructions and degrades throughput.

Methodology: ARCQuant
To address these algorithmic and hardware barriers, the authors propose ARCQuant (Augmented Residual Channels Quantization). The core innovation is a framework that maintains a strictly unified NVFP4 data path while compensating for quantization errors through an augmented residual mechanism.

Adaptive Outlier Identification: Using calibration data, the method identifies dominant outlier channels. It reorders channels based on absolute maximums and selects a top- $S$ subset where the precision gap between the target NVFP4 (E2M1 mantissa) and a reference FP8 format is significant.
Dual-Stage Quantization:
- Primary Stage: The input activation matrix is reordered and quantized using standard block-wise NVFP4 quantization.
- Residual Stage: For the identified outlier channels, the method computes the residual (the difference between the original value and the primary quantized value). These residuals are then quantized separately into a second set of NVFP4 channels.
Augmented Matrix Multiplication: The quantized residuals are concatenated with the primary quantized activations along the reduction dimension ( $K$ ). Similarly, the corresponding weights are duplicated to match this augmentation.
Unified GEMM Execution: The entire process is mapped into a single, extended matrix multiplication operation $(N, K_{in} + S, M)$ . This allows the compensation process to be executed using standard, highly optimized CUTLASS GEMM kernels without modifying the hardware data path or requiring mixed-precision instructions. The linearity of accumulation ensures the primary computation and residual corrections are summed directly into the high-precision output accumulator.
Kernel Implementation: The authors implement a fused CUDA kernel that integrates channel reordering, RMSNorm, primary quantization, and residual quantization. This minimizes latency and outputs data in an interleaved channel layout ready for the GEMM.

Key Contributions

First NVFP4-Optimized PTQ Framework: ARCQuant is the first framework specifically designed for NVFP4 that utilizes augmented residual channels to bypass mixed-precision hardware constraints, achieving W4A8-level accuracy within strict W4A4 constraints.
Theoretical Error Bound Analysis: The authors provide a rigorous analysis showing that the worst-case error bound of their dual-stage NVFP4 mechanism is comparable to standard single-stage MXFP8. Specifically, the alignment overhead of the dual-stage process ( $\approx 1.266$ ) is lower than that of MXFP8 ( $< 2$ ), theoretically validating the bridging of the precision gap.
High-Performance Implementation: The paper details the implementation of custom kernels that map complex residual accumulation into standard GEMM calls, ensuring portability and efficiency on Blackwell architectures.

Experimental Results
Extensive experiments were conducted on LLaMA 3.1 and Qwen2.5 model families (ranging from 7B to 70B) and evaluated on NVIDIA RTX 5090 and RTX PRO 6000 GPUs.

Accuracy: ARCQuant consistently outperforms existing NVFP4 adaptation strategies (including RTN, SmoothQuant, QuaRot, and MicroMix). On LLaMA 3.1-8B and Qwen2.5-7B, it surpasses W4A8 baselines in perplexity and MMLU scores, effectively closing the gap to full FP16 precision. On Qwen2.5-32B, it achieves near-lossless compression.
NVFP4 Specifics: Unlike rotation-based methods which caused performance regression on NVFP4, ARCQuant successfully suppresses outlier errors without disrupting block isolation.
Efficiency: The method achieves significant speedups over FP16 (up to 3.5× on RTX 5090 for Llama 3.1-8B) and memory reductions (1.5×–2.8×). The overhead introduced by the residual augmentation is minimal (approx. 4.9% total latency increase), and it significantly outperforms W4A8 and MXFP8 alternatives in throughput.
Generalizability: The approach demonstrates robustness across different model sizes (including 70B dense and 8x7B MoE architectures) and shows extensibility to INT4 and MXFP4 formats.

Significance and Claims
The paper claims that ARCQuant offers a scalable, hardware-friendly pathway for efficient LLM inference by resolving the outlier challenge without compromising block isolation or hardware uniformity. The authors emphasize the principle of "trading minimal compute dimensions for higher fidelity," enabling the use of aggressive 4-bit quantization on next-generation hardware while maintaining the accuracy of higher-precision formats. The work is presented as a practical solution that leverages the specific capabilities of the NVIDIA Blackwell architecture to overcome the limitations of current PTQ strategies.