Technical Summary: AAAC (Activation-Aware Adaptive Codebooks)

1. Problem Statement

Large Language Models (LLMs) face significant memory and compute constraints, making post-training weight-only quantization to 4 bits (W4A16/W4A8) a critical deployment strategy. Existing Post-Training Quantization (PTQ) methods generally operate under a fixed-grid assumption. They attempt to improve accuracy by optimizing how weights are mapped onto a pre-defined, static 4-bit scalar grid (e.g., standard INT4 or NVFP4 formats) through techniques like channel scaling (AWQ), clipping, or error compensation (GPTQ).

The authors argue that this approach leaves potential quality on the table. While the 4-bit format fixes the number of stored codes and the group-wise scale structure, it does not necessitate that the scalar reconstruction grid itself be fixed. The dequantized values, which determine the actual weights used during inference, are constrained by this fixed table. Current methods improve the mapping to the table but do not adapt the table itself to the specific weight distribution and activation statistics of a given layer.

2. Methodology: AAAC

The paper proposes AAAC (Activation-Aware Adaptive Codebooks), a lightweight, gradient-free method that replaces the single fixed scalar table $T$ with two learned scalar codebooks ($T_0$ and $T_1$) per layer.

Core Mechanism

1. Adaptive Codebooks: Instead of a single fixed grid, AAAC learns two small scalar codebooks per layer. These codebooks have the same cardinality as the base 4-bit format (e.g., 15 entries for NVFP4, 16 for INT4) but their values are optimized from calibration data rather than fixed by the format specification.
2. Activation-Aware Selection: For each group of weights, the method selects the codebook ($T_0$ or $T_1$) that minimizes the activation-weighted reconstruction error. The importance of each weight is determined by the squared activation energy of its corresponding column in the calibration set ($I_k = \sum X_{t,k}^2$).
3. Zero-Overhead Storage:
   • Codebook Size: Each codebook contains only ~15–16 BF16 values, totaling approximately 64 bytes per layer. For a 7B model, this adds only ~14 KB of storage, which is negligible.
   • Selection Bit: The choice of codebook for a group is encoded in the unused sign bit of the group's positive scale factor (since scales are always positive). This allows for per-group selection with zero additional storage overhead when the selection group size ($S$) matches the scale group size ($g$).
4. Learning Algorithm:
   • Initialization: $T_0$ and $T_1$ are initialized using evenly spaced quantiles of the normalized weight distribution, with $T_1$ shifted by a half-step to provide distinct starting points.
   • Alternating Optimization: The algorithm alternates between:
     • Assignment: Assigning each weight group to the codebook that minimizes activation-weighted error.
     • Refinement: Updating codebook entries using activation-weighted k-means (computing the weighted centroid of the Voronoi cell for each entry).
   • Efficiency: The process requires no backpropagation, no Hessian computation, and no full-model gradients. It processes layers independently using only forward-pass activations.

3. Key Contributions

• Redefining the PTQ Objective: AAAC shifts the focus from mapping weights to a fixed grid to learning the grid itself, introducing a new degree of freedom in quantization.
• Lightweight Gradient-Free Optimization: Unlike gradient-assisted methods (e.g., OmniQuant, SqueezeLLM) that require hours of backpropagation and massive memory overhead (2–4× model size), AAAC learns via simple k-means on forward activations.
• Zero Storage Overhead: By leveraging the unused sign bit of the scale factor, AAAC achieves adaptive codebook selection without increasing the model's memory footprint.
• Speed: Quantization completes in 3–30 minutes on a single GPU, orders of magnitude faster than gradient-assisted baselines.

4. Experimental Results

The authors evaluated AAAC across Llama (1B–13B) and Qwen (0.5B–32B) model families using both NVFP4 ($g=16$) and INT4 ($g=128$) layouts.

• vs. Gradient-Free Methods: AAAC consistently outperforms RTN, IF4, AWQ, and GPTQ.
  • On WikiText-2, AAAC achieved a 59.2% gap recovery (vs. RTN) on average for NVFP4, significantly surpassing AWQ (28.1%) and IF4 (12.3%).
  • In INT4 settings ($g=128$), AAAC with zero overhead ($S=128$) already outperformed AWQ on most models; adding a negligible 0.0625 bpw overhead ($S=16$) further improved performance.
• vs. Gradient-Assisted Methods: AAAC matches or exceeds the performance of methods requiring hours of optimization and high memory.
  • AAAC achieved 60.0% average gap recovery, matching QuIP# with fine-tuning (60.0%) and outperforming OmniQuant (55.0%), SqueezeLLM (55.0%), and GPTVQ (25.0%).
  • Crucially, AAAC achieves this with peak memory no larger than the original model size and in minutes rather than hours.
• Downstream Tasks: On 8 downstream tasks (reasoning, knowledge, commonsense, math) using Qwen3.5 models, AAAC achieved the highest average accuracy (64.21%) among all quantized methods, recovering 49.3% of the gap to the BF16 baseline.
• Combination with AWQ: When combined with AWQ (scaling weights before quantization), AAAC recovers 70.5% of the quantization gap on average, demonstrating compatibility with other PTQ techniques.

5. Significance and Claims

The paper claims that AAAC challenges the prevailing assumption that gradients are necessary to close the quality gap between 4-bit quantized models and full-precision baselines. By demonstrating that a lightweight, gradient-free method can match or exceed the performance of complex, gradient-assisted algorithms, the authors suggest that learned codebooks at a minimal scale represent a practical and efficient direction for 4-bit LLM quantization.

The significance lies in the trade-off: AAAC offers near-state-of-the-art accuracy (rivaling methods that require 100 GPU-hours) with minimal computational cost (minutes) and zero memory overhead, making high-quality 4-bit quantization accessible for a wider range of deployment scenarios.

Limitations noted by the authors:

• Evaluation is limited to models up to 32B parameters; 70B+ dynamics were not tested.
• Comparisons with gradient-assisted methods are restricted to LLaMA 7B/13B due to protocol consistency in prior literature.
• The method targets weight-only quantization (W4A16/W4A8) and does not address joint weight-activation quantization (W4A4).