Technical Summary: Preserve-Then-Quantize (SRR)

Problem Statement

Post-Training Quantization (PTQ) is a primary method for reducing the memory footprint and inference costs of Large Language Models (LLMs) by converting full-precision weights to low-bit representations. However, aggressive low-bit quantization (e.g., 2-bit or 3-bit) often causes substantial accuracy degradation. A widely adopted remedy is Quantization Error Reconstruction (QER), which approximates the full-precision weight matrix $W$ as $W \approx Q + LR$, where $Q$ is the quantized weight and $LR$ is a low-rank correction term designed to recover information lost during quantization.

Existing QER methods typically devote the entire rank budget $r$ to reconstructing the quantization residual (i.e., approximating $S(W-Q)$, where $S$ is an activation-scaled matrix). The authors argue this approach is suboptimal for two main reasons:

1. Inefficient Resource Allocation: In low-bit regimes, quantization error is often dense and high-rank. Devoting all rank capacity to correcting this error can be inefficient.
2. Structural Damage: Transformer weights in the activation-scaled space ($SW$) are often highly anisotropic, with most energy concentrated in a small set of dominant singular directions. Standard QER quantizes these dominant directions first, injecting disproportionately large scaled errors. The subsequent low-rank correction is then forced to spend its limited capacity repairing "avoidable distortion" in the most informative subspace rather than correcting the residual noise.

The central question addressed is: Under a fixed rank budget, how should we balance the preservation of dominant structure versus the reconstruction of quantization error?

Methodology: Structured Residual Reconstruction (SRR)

The paper proposes Structured Residual Reconstruction (SRR), a "preserve-then-quantize" framework that explicitly allocates the rank budget between two roles: preserving dominant structure and reconstructing quantization error.

1. The Preserve-Then-Quantize Pipeline

Instead of quantizing the entire weight matrix $W$ and then correcting the residual, SRR performs the following steps for a chosen split $k$ (where $0 \le k \le r$):

1. Preserve: Extract the top-$k$ dominant singular directions of the scaled weight matrix $SW$. These directions are mapped back to the original space ($L^{(1)}_k R^{(1)}_k$) and bypass quantization.
2. Quantize: The remaining residual component ($W - L^{(1)}_k R^{(1)}_k$) is quantized to produce $Q_k$.
3. Reconstruct: The quantization error induced on the residual ($E_k = (W - L^{(1)}_k R^{(1)}_k) - Q_k$) is reconstructed using the remaining rank budget $r-k$ via truncated SVD in the scaled space ($L^{(2)}_k R^{(2)}_k$).

The final approximation is $\hat{W}_{SRR} = L^{(1)}_k R^{(1)}_k + Q_k + L^{(2)}_k R^{(2)}_k$.

2. Theory-Guided Rank Selection ($k^*$)

Selecting the optimal split point $k$ is critical. SRR derives a criterion to minimize the scaled reconstruction error without expensive enumeration.

• Objective: Minimize $\|S(W - \hat{W}_{SRR}(k))\|_F$.
• Derivation: The error is factorized into the scale of the quantization error and the unrecoverable energy ratio.
  • Assumption 4.1: The scaled quantization error energy is approximately proportional to the scaled input energy ($\|SE_k\|_F \approx \eta_Q \|S(W - L^{(1)}_k R^{(1)}_k)\|_F$).
  • Assumption 4.2: The spectral profile of the normalized quantization error is insensitive to the specific realization and can be approximated by a random matrix $E$ with i.i.d. entries.
• Selection Rule: The optimal $k^*$ is selected by minimizing the product of the unrecoverable energy of the preserved part and the unrecoverable energy of the residual part:
  $$k^* = \arg \min_{0 \le k \le r} \rho_k(SW) \cdot \rho_{r-k}(SE)$$
  where $\rho_p(A)$ is the ratio of energy remaining after a rank-$p$ truncation. This allows for a lightweight, one-shot selection of $k$ using a random probe matrix $E$.

3. Application to Quantized Parameter-Efficient Fine-Tuning (QPEFT)

SRR extends naturally to QPEFT, where the quantized backbone is frozen, and only the low-rank adapter is trained.

• Initialization: The SRR decomposition provides a high-fidelity initialization for the adapter ($LR = L^{(1)}R^{(1)} + L^{(2)}R^{(2)}$).
• Gradient Scaling: The two components play different roles: $L^{(1)}R^{(1)}$ preserves the dominant subspace, while $L^{(2)}R^{(2)}$ handles adaptation. To prevent the optimizer from over-updating the preserved directions (which could destabilize the structure), SRR applies gradient scaling to the preserved component:
  $$\nabla_{L^{(1)}R^{(1)}} L \leftarrow \gamma \nabla_{L^{(1)}R^{(1)}} L, \quad \gamma \in (0, 1)$$
  This stabilizes fine-tuning by attenuating updates along the preserved subspace while allowing the residual directions to adapt freely.

Key Contributions

1. Rank-Allocation Framework: SRR introduces a principled framework that explicitly splits the rank budget between preserving dominant low-rank structure and reconstructing quantization error, addressing the inefficiency of "residual-only" reconstruction.
2. Theory-Guided Selection: The paper derives a simple, theory-guided criterion for selecting the split point $k$ based on the spectral decay of the scaled weights and a random-matrix proxy for quantization noise, avoiding expensive enumeration.
3. QPEFT Stabilization: The method provides a structured initialization for QPEFT and introduces a gradient scaling mechanism to stabilize training by decoupling the roles of preserved and reconstructed directions.
4. Plug-and-Play Implementation: SRR is designed as a fine-tuning-free, plug-and-play module compatible with existing activation-aware QER methods (e.g., LQER, QERA).

Experimental Results

The authors evaluated SRR across diverse models (TinyLLaMA, Gemma-2, LLaMA-2, LLaMA-3.1) and quantization settings (2-bit to 4-bit).

• Post-Training Quantization (PTQ):

  • SRR consistently reduced perplexity on WikiText2 across all models and rank budgets ($r=32, 64$) compared to baselines like LQER and QERA.
  • Notable improvements included a 27.1% relative perplexity reduction on LLaMA-2 7B and 12.2% on Gemma-2 2B at 3-bit precision.
  • Zero-shot accuracy on downstream tasks (HellaSwag, Winogrande, etc.) improved consistently.

• Quantized PEFT (QPEFT):

  • On GLUE benchmarks using RoBERTa-base, SRR achieved a 5.9 percentage-point average gain over the QERA baseline under 2-bit quantization.
  • SRR outperformed iterative methods like LoftQ and LQ-LoRA by over 10 percentage points at 2-bit precision without requiring iterative refinement.
  • Training loss curves demonstrated that SRR accelerates convergence compared to baselines.
  • On GSM8K and SlimPajama, SRR improved accuracy and reduced perplexity significantly compared to QERA and other baselines at 2-bit precision.

• Robustness:

  • SRR showed consistent improvements across different quantizers (MXINT, GPTQ, QuIP#).
  • The rank selection $k^*$ was found to be stable across random seeds (typically varying by $\pm 1$).

Significance and Claims

The paper claims that balancing preservation and reconstruction is critical for accurate low-bit approximation of $W \approx Q + LR$. By preventing quantization from corrupting high-energy directions and reserving low-rank capacity for the most effective corrections, SRR achieves a more accurate approximation than prior methods under identical rank budgets.

The authors emphasize that SRR offers a principled framework for low-rank refinement in quantized models. It demonstrates that the low-rank budget should not serve a single purpose (error correction) but should be dynamically allocated based on the intrinsic structural properties of the weights. The method is presented as a robust, computationally efficient (negligible overhead via randomized SVD), and generalizable solution for both PTQ and QPEFT, enabling more accurate low-bit deployment of LLMs.