💻 computer science

Sensitivity-aligned Mixed-precision Quantization and Low-rank Adaptation for LLMs

This paper proposes a sensitivity-guided framework that combines block-wise mixed-precision quantization with adaptive low-rank adaptation to efficiently compress large language models while preserving or even improving downstream performance and perplexity compared to FP16 baselines.

Original authors: Syed Faizan Hussain Shah, Gyeongmin Kim

Published 2026-10-16

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Syed Faizan Hussain Shah, Gyeongmin Kim

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Large language models are the engines behind the most advanced artificial intelligence systems today, capable of writing stories, solving complex problems, and holding conversations that feel remarkably human. These systems are built on massive networks of mathematical connections, often containing billions of parameters, which allow them to learn patterns from vast amounts of text. However, this intelligence comes with a heavy price: the sheer size of these models requires enormous amounts of computer memory and processing power. This makes them difficult to run on everyday devices like smartphones or laptops, and even expensive data centers struggle with the energy and speed demands of such large systems. To make these powerful tools more accessible, researchers have spent years developing ways to shrink them without losing their ability to think clearly.

One of the most promising methods for shrinking these models is called quantization. Imagine a library where every book is written in a high-resolution, full-color format that takes up a lot of shelf space. Quantization is like reprinting those books in a smaller, more compact font or using fewer colors to save space. In the world of computers, this means representing the numbers inside the model with fewer bits of information. While this significantly reduces the size of the model, it often introduces errors, much like a photocopy that loses some detail. If the compression is too aggressive, the model's ability to understand language can degrade, leading to nonsensical or incorrect answers. The challenge has been finding a way to compress the model enough to fit on smaller devices while keeping the quality of its output high.

A new study by researchers at Jeju National University proposes a clever solution to this problem by treating different parts of the model differently, rather than applying a single compression rule to the whole thing. They observed that not all parts of a language model are equally important. Some sections are highly sensitive to changes and act as the critical backbone of the model's reasoning, while other sections are more robust and can tolerate being compressed more heavily. The researchers developed a system that maps out these sensitive areas and assigns them a higher precision, keeping them detailed and clear, while allowing the less sensitive areas to be compressed into a smaller, more efficient format. This approach is similar to how a photographer might keep the face of a subject in sharp focus while slightly softening the background to save space, ensuring the most important details remain intact.

The researchers, Syed Faizan Hussain Shah and Gyeongmin Kim, created a framework they call A2BQ, which stands for attention-derived block-wise mixed-precision quantization. Instead of looking at the model as a single block of data, they broke it down into smaller chunks. They then analyzed how much each chunk relied on the flow of information within the model, using a signal derived from how the model pays attention to different words. Based on this analysis, they assigned different levels of precision to each chunk. Some chunks were kept at a high level of detail, using eight bits of information, while others were compressed to six or even four bits. This created a mixed-precision map where the most critical parts of the model remained robust, and the less critical parts were shrunk down to save space.

However, simply compressing the model, even in a smart way, still leaves some room for error. To fix this, the researchers added a second step called adaptive low-rank adaptation, or qLoRA+. Think of this as a fine-tuning process where the model is given a chance to learn how to correct the small mistakes introduced by the compression. In previous methods, this fine-tuning was applied uniformly across the entire model, giving every part the same amount of help. The new approach is more strategic: it gives more "learning capacity" to the parts of the model that were compressed the most or were identified as being the most sensitive. By directing the model's ability to learn specifically toward the areas that need it most, the researchers were able to recover much of the quality that might have been lost during compression.

The team tested this combined approach on five different large language models, ranging in size from 1.1 billion to 8 billion parameters. They measured how well these models performed on standard language tasks, such as predicting the next word in a sentence, which is a key indicator of how well a model understands language. The results showed that their method consistently outperformed models that were compressed using a single, uniform precision level. In fact, for the smallest model they tested, the new method achieved a specific score of 7.884 on a standard test, which was an improvement over the corresponding mixed-bit baseline, while also running at a speed of 73.11 tokens per second. This suggests that by carefully balancing where to compress and where to keep detail, and then helping the model learn to adapt to those changes, it is possible to make these powerful systems smaller without sacrificing their intelligence.

One of the most important findings of the study concerns the difference between how much space a model theoretically saves and how much memory it actually uses while running. The researchers noted that while their method could theoretically reduce the storage size of the model by about 4 to 8 percent, the actual memory used during operation remained close to that of the original, uncompressed model. This is because their current system reconstructs the compressed data back into a standard format before the computer processes it. While this means the immediate speed benefits of having a smaller file on the disk are not yet fully realized in their current setup, the study proves that the strategy of mixing different precision levels and adapting the model accordingly works to preserve quality. Furthermore, the researchers found that while the method improved language modeling scores, it did not guarantee uniform improvements across all downstream tasks; performance on specific benchmarks remained generally close to the original uncompressed models, with some tasks showing slight decreases. The researchers argue that this approach offers a clear path forward: by understanding which parts of a model are fragile and which are sturdy, we can design systems that are both efficient and intelligent, paving the way for more capable artificial intelligence to run on the devices we use every day.

Technical Summary: Sensitivity-aligned Mixed-precision Quantization and Low-rank Adaptation for LLMs

1. Problem Statement

Large Language Models (LLMs) face significant deployment challenges due to their massive memory footprints and high inference-time bandwidth requirements, particularly on resource-constrained platforms like edge devices and single-GPU systems. While model compression techniques such as pruning, distillation, and quantization offer solutions, standard Post-Training Quantization (PTQ) methods often apply uniform bit-widths (e.g., all 4-bit or all 8-bit) across the entire network. This uniform approach overlooks the heterogeneity of transformer layers; empirical evidence suggests that quantizing sensitive layers disproportionately harms accuracy, especially at low bit-widths.

Furthermore, existing mixed-precision strategies typically operate at a coarse "layer-wise" granularity, assigning a single bit-width per layer based on aggregated metrics. This fails to capture intra-layer variability (e.g., differences across attention heads or feed-forward blocks). Additionally, while Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA and qLoRA enable adaptation with minimal parameters, they generally assume uniform adapter ranks across all layers. This assumption is suboptimal in mixed-precision settings where quantization severity and representational capacity vary significantly across the network depth. The paper addresses the lack of an explicit connection between fine-grained mixed-bit precision planning and the configuration of adaptive low-rank adaptation.

2. Methodology

The authors propose a sequential framework coupling Attention-Derived Block-wise Quantization (A2BQ) with qLoRA+, a plan-guided low-rank adaptation strategy. The framework operates in three main stages:

2.1 Attention-Derived Block-Wise Sensitivity Estimation (A2BQ)

Instead of layer-wise sensitivity, the method estimates sensitivity at the block granularity (row blocks of fixed size, e.g., 128 rows).

Signal: Sensitivity is derived from the variance of post-softmax attention probabilities on a calibration corpus. High variance in attention distributions indicates a block is sensitive to quantization noise.
Aggregation: Block-level sensitivity scores are computed by averaging the standardized attention-variance scores of the attention heads intersecting the block's hidden-coordinate range.

2.2 Constrained Mixed-Bit Allocation

Using the sensitivity map, the framework assigns 4-bit, 6-bit, or 8-bit precision to each block under a global average-bit budget (e.g., 5.7 bits).

Optimization: A greedy cost-risk procedure minimizes a sensitivity-weighted penalty. Blocks with higher sensitivity scores are protected with higher precision (8-bit), while less sensitive blocks are aggressively compressed (4-bit).
Granularity: This allows for heterogeneous precision within a single transformer layer, capturing intra-layer variability that layer-wise methods miss.

2.3 Quantization-Aware Adaptive Low-Rank Adaptation (qLoRA+)

The framework introduces qLoRA+, which conditions the rank of LoRA adapters on both the estimated quantization sensitivity and the allocated precision of each layer.

Rank Allocation: The adaptive rank $r_i$ $r_{i}$ for layer $i$ $i$ is computed as a function of the layer's normalized sensitivity ( $\hat{S}_i$ $\hat{S}_{i}$ ) and its average allocated precision ( $\bar{b}_i$ $\overset{ˉ}{b}_{i}$ ).
- Layers that are more sensitive and more aggressively quantized (lower bit-width) receive higher adapter ranks to provide greater recovery capacity.
- This creates an explicit link between the precision plan and the adaptation capacity.
Training Pipeline:
1. The base model is frozen in 4-bit NF4 format.
2. qLoRA+ adapters are trained using the sensitivity-guided ranks.
3. Adapters are merged into the original FP16 pretrained weights.
4. The merged model undergoes the final mixed-bit rewriting (A2BQ) for deployment.

2.4 Deployment and Execution

The final model is rewritten according to the mixed-bit allocation map. The authors employ a dequantized-buffer (DQ) reference backend:

Quantized weights are reconstructed into floating-point buffers before inference.
Forward computation uses standard floating-point linear operations.
This distinction is crucial: the "packed payload" represents theoretical storage savings, while runtime memory reflects the dequantized execution.

3. Key Contributions

A2BQ Framework: An attention-derived, block-wise mixed-precision allocation framework that uses post-softmax attention variance to rank hidden-row blocks and assigns 4/6/8-bit precision under an average-bit constraint.
qLoRA+ Strategy: A plan-guided adaptation strategy that explicitly links mixed-bit precision planning to adaptation configuration by conditioning layer-wise adapter ranks on both sensitivity and allocated precision.
Sequential Pipeline: A deployment pipeline where adapters are trained on a frozen NF4 backbone, merged, and then subjected to mixed-bit rewriting with explicit module eligibility and boundary-protection constraints.
Empirical Analysis: A comprehensive evaluation across five model families (1.1B to 8B parameters) distinguishing theoretical packed-weight payload from runtime memory behavior, and analyzing layer-wise KL divergence to characterize distribution shifts.

4. Experimental Results

The framework was evaluated on models including TinyLlama-1.1B, Qwen2.5-3B, Phi-3-mini-4k, Mistral-7B, and LLaMA-3.1-8B using WikiText-2 perplexity and downstream benchmarks (MMLU, GSM8K, etc.).

Perplexity Improvement: Combining A2BQ with qLoRA+ consistently outperformed mixed-bit rewriting alone across all model families.
- Example (TinyLlama-1.1B): Mixed-bit only achieved 8.994 PPL, while A2BQ + qLoRA+ achieved 7.884 PPL, surpassing the FP16 baseline (10.440) and standard qLoRA with fixed rank (8.688). The proposed qLoRA+ method alone achieved 8.036 PPL.
- Similar improvements were observed for Qwen2.5-3B (9.077 $\to$ 8.335), Phi-3-mini-4k (7.884 $\to$ 6.922), Mistral-7B (6.938 $\to$ 6.571), and LLaMA-3.1-8B (8.258 $\to$ 7.501).
Inference Efficiency: The proposed method generally maintained comparable or improved decoding throughput compared to mixed-bit-only baselines. For TinyLlama-1.1B, throughput increased from 60.60 tok/s (mixed-bit only) to 73.11 tok/s.
Memory vs. Storage:
- Theoretical Payload: The estimated packed parameter payload reduction ranged from 4.02% to 7.90% across models. This is lower than global 4-bit PTQ methods (which reduce payload by ~65-72%) because the current implementation only rewrites eligible projections (e.g., Q/O attention, not K/V in grouped-query attention).
- Runtime Memory: Due to the dequantized-buffer backend, measured peak GPU memory remained close to the FP16 reference (e.g., ~2.08 GB for TinyLlama), demonstrating that the current implementation does not yet realize physical memory bandwidth savings from packed low-bit execution.
Downstream Tasks: Performance on downstream benchmarks (MMLU, GSM8K, TruthfulQA, ARC-Challenge) remained largely stable or showed slight improvements compared to FP16 baselines, indicating that the method preserves task-specific capabilities.
Distribution Shift: Layer-wise KL divergence analysis revealed that quantization-induced shifts are non-uniform across depth. While sensitivity-guided allocation aligns with precision assignment, the correlation between sensitivity and post-quantization KL divergence is weak and architecture-dependent, suggesting sensitivity is an allocation signal rather than a perfect predictor of distributional shift.

5. Significance and Claims

The paper claims that sensitivity-guided mixed-bit allocation combined with adaptive low-rank adaptation offers an effective approach for balancing model quality, precision allocation, and deployment-oriented efficiency.

Quality Recovery: The primary contribution is demonstrating that adaptive rank allocation can recover model quality lost to mixed-precision rewriting without requiring a separate adapter payload in the final merged configuration.
Granularity: The work highlights the importance of block-level granularity over layer-level granularity in capturing intra-layer sensitivity variations.
System Awareness: The authors emphasize the critical distinction between algorithmic compression potential (theoretical packed storage) and runtime implementation behavior (dequantized execution). They argue that current mixed-precision evaluations must clearly separate these metrics to avoid conflating storage savings with actual inference memory or speedups.
Modest Scope: The paper acknowledges that the current rewrite scope is limited to specific projections (due to eligibility constraints) and that the backend does not yet utilize native packed low-bit kernels. Therefore, the reported throughput and memory metrics reflect the reference implementation's behavior rather than the theoretical maximum of a fully optimized hardware backend. The framework is presented as a step toward realizing full storage and hardware efficiency potential through future work on broader rewrite scopes and native kernels.