Technical Summary: RaZeR – Pushing the Limits of NVFP4 Quantization with Redundant Zero Remapping

1. Problem Statement

Large Language Models (LLMs) face significant challenges in deployment due to substantial memory bandwidth and computational demands. While quantization is a primary solution for efficient inference, the recently introduced NVFP4 format (supported by NVIDIA Blackwell GPUs) presents a specific limitation: the restricted set of representable values in the 4-bit FP4 element format.

The authors identify two specific sources of redundancy in the current NVFP4 encoding that are underutilized:

1. Redundant FP4 Zeros: The FP4 element format (FP4-E2M1) uses a sign-magnitude representation, resulting in both positive and negative zeros. One of these zeros is redundant for representing numerical values.
2. Redundant Block Scaling Sign Bit: The block scaling factor in NVFP4 is an FP8-E4M3 value. However, since scaling factors are derived from absolute maximums of positive tensors, they are always positive. Consequently, the sign bit of the FP8 scaling factor is unused, effectively wasting 1 bit of precision per block (or 2 bits for weights where a more compact scale format is viable).

These redundancies limit the accuracy of 4-bit quantization, particularly for weight-only and weight-activation scenarios, as the standard FP4 set $\{ \pm 0, \pm 0.5, \pm 1, \pm 1.5, \pm 2, \pm 3, \pm 4, \pm 6 \}$ cannot optimally cover the distribution of LLM weights and activations.

2. Methodology: Redundant Zero Remapping (RaZeR)

The paper proposes RaZeR, an enhanced numerical format that repurposes these redundancies to increase quantization accuracy without increasing memory footprint.

2.1. Exploiting Scaling Factor Redundancy

The authors first analyze the bit-width requirements for block scaling factors. Through empirical evaluation on Llama and Qwen models, they find that LLM weights are tolerant to reduced-precision scaling factors.

• Weights: The block scale can be reduced from the standard E4M3 (8 bits) to E3M3 (6 bits) without accuracy loss, freeing up 2 bits per block.
• Activations: Activations have a larger dynamic range and are less tolerant; thus, the authors retain the E4M3 (7 bits effective, 1 sign bit redundant) format, freeing up 1 bit per block.

2.2. Remapping the Redundant Zero

RaZeR utilizes the freed bits from the scaling factor to encode metadata that selects a "Special Value" to replace the redundant zero in the FP4 element representation.

• Special Value Selection: Instead of representing a zero, the redundant FP4 encoding is remapped to a specific non-zero value (e.g., $\pm 5$).
• Optimization: For each block, the algorithm selects the optimal special value from a pre-defined set to minimize the mean squared quantization error.
  • Weights: With 2 free bits, RaZeR can select from 4 special values (2 pairs). These are pre-computed offline.
  • Activations: With 1 free bit, RaZeR selects from 2 special values (1 pair). This requires a lightweight online quantization step (double quantization) to compare errors, adding less than 2% overhead to the quantization time.
• Value Constraints: To maintain hardware efficiency, special values are restricted to multiples of 0.5 (matching FP4 granularity). Empirical analysis shows that $\pm 5$ is the optimal special value pair, as it bridges the gap between the largest FP4 representable values ($\pm 4$ and $\pm 6$).

2.3. Implementation

• GPU Kernels: The authors implemented high-performance weight-only and weight-activation GEMM kernels for NVIDIA Blackwell GPUs. They utilize a two-pass method for activations to handle the remapping logic within existing tensor cores and employ auto-tuning strategies to optimize thread block partitioning for small batch sizes.
• Hardware Design: A modified NVFP4 tensor core is proposed to natively support RaZeR. This design includes weight and activation decoders that use offset registers to reconstruct the special values from the metadata bits before performing Multiply-Accumulate (MAC) operations.

3. Key Contributions

1. RaZeR Format: An enhanced NVFP4 format that repurposes redundant zero bits and scaling factor sign bits to introduce adaptive special values, improving quantization accuracy at the same memory footprint.
2. Efficient GPU Kernels: Implementation of weight-only and weight-activation inference kernels on Blackwell GPUs that achieve performance comparable to or better than prior 4-bit kernels (e.g., Marlin, AWQ).
3. Hardware Architecture: A design for a modified tensor core that adds native support for RaZeR with minimal silicon overhead.
4. Comprehensive Evaluation: Extensive benchmarks demonstrating RaZeR's superiority over existing 4-bit methods across various model sizes (Llama, Qwen) and quantization settings.

4. Experimental Results

The paper evaluates RaZeR against state-of-the-art methods including NVFP4, FourOverSix, MR-GPTQ, and NF4.

• Perplexity Reduction:
  • Relative to native NVFP4, RaZeR reduces average perplexity loss by 34.6% (weight-only) and 31.2% (weight-activation).
  • Compared to the state-of-the-art FourOverSix, RaZeR reduces average perplexity loss by 29.2% (weight-only) and 23.3% (weight-activation).
• Zero-Shot Accuracy: RaZeR improves average zero-shot task accuracy by 0.63% over the baseline NVFP4 format across all tested models.
• Reasoning Tasks: On GSM8K and MMLU tasks, RaZeR shows significant gains, outperforming recent NVFP4-based algorithms (MR-GPTQ, FourOverSix) by margins up to 4.47% in weight-activation settings.
• Hardware Efficiency:
  • Area/Power: Synthesis results on TSMC 28nm technology show the RaZeR tensor core incurs only 3.7% area and 13.5% power overhead relative to the baseline NVFP4 core. Given MAC units are a small fraction of total accelerator design, the total chip overhead is estimated at 0.37% area and 1.35% power.
  • Throughput: On Blackwell GPUs, RaZeR kernels achieve near-best-in-class single-batch decoding speeds and substantially outperform other weight-only kernels at moderate to large batch sizes.

5. Significance and Claims

The paper claims that RaZeR represents a significant step forward in low-precision LLM quantization by demonstrating that algorithmic innovation can be achieved by exploiting structural redundancies in existing hardware-supported formats rather than requiring entirely new, complex data types.

Key claims include:

• RaZeR pushes the limits of NVFP4, achieving higher accuracy than current state-of-the-art 4-bit methods without increasing memory usage.
• The approach is practical for immediate deployment on current hardware (via software kernels) and future hardware (via the proposed tensor core modification).
• The minimal hardware overhead (sub-1% total chip area) makes the adoption of RaZeR highly feasible for next-generation accelerators.
• By reducing the operational cost and energy consumption of LLM deployment through more efficient quantization, the work contributes to making AI systems more accessible.

The authors emphasize that their work focuses on refining the NVFP4 standard to better suit LLM tensor distributions, offering a path to "near-lossless" 4-bit quantization that was previously unattainable with standard FP4 or integer formats.