Technical Summary: QWHA - Quantization-Aware Walsh-Hadamard Adaptation for Parameter-Efficient Fine-Tuning on Large Language Models

1. Problem Statement

The deployment of Large Language Models (LLMs) faces a dual challenge: reducing inference costs through quantization and lowering training overhead via Parameter-Efficient Fine-Tuning (PEFT). While Quantization-Aware PEFT (QA-PEFT) aims to combine these benefits, existing approaches face significant limitations.

Current QA-PEFT methods predominantly rely on Low-Rank Adaptation (LoRA). While effective, LoRA suffers from limited representational capacity due to its strict rank constraints. Recent alternatives utilizing Fourier-related Transforms (FT) offer superior representational power by training sparse coefficients in the transform domain. However, directly applying FT-based adapters to quantized models often yields inferior performance compared to LoRA-based QA-PEFT. This is because:

1. Initialization Failure: Existing FT-based adapters lack effective initialization strategies to compensate for quantization errors prior to fine-tuning. Identifying the optimal sparse set of parameters to approximate a matrix is an NP-hard problem, and random or magnitude-based selection often fails to capture critical outlier-induced errors.
2. Computational Overhead: Conventional FT-based adapters (e.g., using DFT, DCT, or DHT) require double transforms (input and output dimensions) and complex kernel operations, leading to significant computational costs that negate the efficiency gains of quantization.
3. Outlier Sensitivity: LLM quantization errors are heavily influenced by outliers (weights with exceptionally large magnitudes). Standard sinusoidal bases used in DCT/DHT are smooth and struggle to represent the abrupt transitions of these outliers efficiently.

2. Methodology: QWHA

The authors propose QWHA (Quantization-Aware Walsh-Hadamard Adaptation), a framework that integrates a Walsh-Hadamard Transform (WHT)-based adapter with a novel quantization-aware initialization scheme.

2.1 Adapter Design: Walsh-Hadamard Adaptation (WHA)

Unlike conventional FT-based adapters that apply transforms to both input and output dimensions ($\Delta W = H'^{-1} F H^{-1}$), QWHA employs a single-transform formulation:
$$\Delta W = F H^{-1}$$
Where:

• $H$ is a predefined Walsh-Hadamard Transform (WHT) matrix.
• $F$ is a trainable sparse coefficient matrix constructed from a value vector $c$ and an index list $E$.
• $H^{-1}$ is the inverse WHT.

Key Advantages of WHT:

• Energy Concentration: The WHT kernel consists solely of $\pm 1$ entries, forming square-wave basis functions with sharp transitions. This structural property aligns better with the heavy-tailed distribution of quantization errors (outliers) compared to the smooth sinusoidal bases of DCT or DHT. Empirical analysis shows WHT concentrates the majority of error energy in fewer coefficients, enabling accurate reconstruction with limited parameters.
• Computational Efficiency: The WHT kernel allows for fast recursive computation using only additions and subtractions, eliminating matrix multiplications. This significantly reduces computational overhead compared to DCT/DHT, which often require complex operations or handling of imaginary components.

2.2 Quantization-Aware Initialization

To address the initialization gap in FT-based adapters, QWHA introduces a two-stage initialization strategy: AdaAlloc (Adaptive Allocation) and Refinement.

1. AdaAlloc (Parameter Selection):

   • Instead of random selection or simple magnitude-based selection (which can lead to low-rank matrices by clustering parameters in few channels), AdaAlloc allocates the parameter budget $p$ across output channels proportionally to their activation error magnitudes.
   • This ensures that channels with larger quantization errors receive more parameters, while maintaining a lower bound on parameters per channel to guarantee a full-rank adapter structure.
   • Within each channel, the top-magnitude coefficients are selected to effectively target the error.

2. Refinement (Value Assignment):

   • After selecting the parameter locations, the values are not simply taken from a dense solution. Instead, a least-squares problem is solved to refine the values ($c$) based on the selected basis vectors.
   • This step accounts for the interactions between selected basis vectors, minimizing the layer output error ($\|\Delta W_Q X - F H^{-1} X\|_F^2$) more effectively than direct value assignment.

3. Key Contributions

• WHA Adapter: Introduction of a single-transform, WHT-based adapter that offers higher representational capacity than LoRA and superior outlier handling compared to DCT/DHT-based adapters.
• AdaAlloc & Refinement: A tractable initialization scheme that solves the NP-hard sparse approximation problem by adaptively allocating parameters to error-prone channels and refining values to minimize initialization error.
• Computational Efficiency: Demonstration that the WHT kernel and single-transform design substantially reduce training and initialization latency compared to existing FT-based adapters, achieving speeds comparable to LoRA.

4. Experimental Results

The authors evaluated QWHA on LLaMA-3.1-8B, LLaMA-3.2-3B, and Mistral-7B-v0.3 models across CSQA (commonsense QA) and GSM8k (mathematical reasoning) benchmarks under 2-bit, 3-bit, and 4-bit quantization.

• Accuracy: QWHA consistently outperforms state-of-the-art baselines, including LoRA-based QA-PEFT (CLoQ) and sparse/FT-based adapters (SHiRA, LoCA, SSH). The performance gap is most pronounced in low-bit settings (2-bit and 3-bit), where QWHA achieves 2-3% higher accuracy than baselines.
• Initialization Efficacy: Ablation studies confirm that both the WHT kernel and the AdaAlloc initialization are critical. Without quantization-aware initialization, FT-based adapters perform worse than LoRA. With QWHA's initialization, they significantly outperform LoRA.
• Efficiency:
  • Training Speed: QWHA achieves significant training speedups over existing FT-based adapters (e.g., SSH, LoCA) due to the fast WHT kernel and single-transform design. In some configurations, it is comparable to LoRA.
  • Inference: QWHA incurs minimal inference overhead (1.9% throughput drop compared to LoRA) due to the efficient WHT implementation, whereas other FT-based adapters suffer from substantial overhead (approx. 50% drop).
  • Memory: QWHA reduces total memory consumption by ~13% compared to CLoQ during inference, primarily due to the sparse nature of the adapter and efficient scatter operations.

5. Significance and Claims

The paper claims that QWHA represents a significant advancement in the field of efficient LLM adaptation by successfully bridging the gap between high representational capacity and quantization robustness.

• Overcoming Representational Limits: The work demonstrates that FT-based adapters, when properly initialized and designed with WHT, can surpass the representational limits of low-rank adapters without sacrificing efficiency.
• Addressing Quantization Errors: The study highlights that explicit consideration of quantization effects (specifically outliers) during the initialization phase is crucial for QA-PEFT. The proposed initialization scheme effectively mitigates the accuracy degradation typically associated with aggressive quantization.
• Practical Viability: By reducing the computational overhead of FT-based adapters to levels comparable with LoRA, QWHA makes high-capacity, quantization-aware fine-tuning a practical option for resource-constrained deployment scenarios.

The authors conclude that QWHA provides a robust, efficient, and accurate framework for fine-tuning quantized LLMs, offering a viable alternative to the dominant LoRA paradigm.