🤖 machine learning

MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

MXSens is a training-free, sensitivity-aware mixed-precision quantization method that assigns variable mantissa bitwidths (4/6/8) to LLM weights based on column- and layer-wise sensitivity, effectively mitigating outlier-induced accuracy degradation while leveraging hardware-efficient microscaling formats to outperform existing state-of-the-art baselines.

Original authors: Simla Burcu Harma, Danila Mishin, Zhengyuan Su, Ayan Chakraborty, Elizaveta Kostenok, Dongho Ha, Babak Falsafi, Martin Jaggi, Yunho Oh, Amir Yazdanbakhsh

Published 2026-07-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Simla Burcu Harma, Danila Mishin, Zhengyuan Su, Ayan Chakraborty, Elizaveta Kostenok, Dongho Ha, Babak Falsafi, Martin Jaggi, Yunho Oh, Amir Yazdanbakhsh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to fit a massive, intricate library of knowledge into a tiny, portable backpack. This is the daily challenge for the super-smart computer brains known as Large Language Models (LLMs). These models are brilliant at writing stories, solving riddles, and chatting with us, but they are incredibly heavy, requiring huge amounts of memory and energy to run. To make them light enough to carry around, scientists use a trick called "quantization." Think of this like compressing a high-definition movie into a smaller file size. Instead of storing every single, perfect shade of color, you round them off to the nearest available color on a limited palette. This makes the file tiny and fast to play, but if you round off too much, the picture gets blurry and the details get lost.

The trouble is that these computer brains have a few "loud" voices that scream much louder than everyone else. In the world of numbers, these are called "outliers." They are rare, extreme values that throw off the whole system, making the compression messy and causing the model to make silly mistakes. For a while, researchers tried to fix this by shuffling the data around (like rotating a puzzle) or by using a mix of big and small file sizes. But these methods were often clunky, requiring the computer to constantly stop and translate numbers back and forth, which slowed everything down. Now, a new team of researchers has proposed a smarter way to pack the backpack, one that listens to the specific needs of every single piece of information without slowing the trip down.

The paper introduces a new method called MXSens, which acts like a very attentive librarian. Instead of treating every book in the library the same way, MXSens first takes a quick look to see which books are the most fragile or important. It discovered that not all "loud" voices are created equal. Some are rare, screaming extremes that need a lot of space to be understood clearly, while others are just slightly louder than the crowd and can get by with a little less room.

MXSens uses a clever three-tier system to handle this. It assigns the most sensitive, extreme parts of the model a generous 8-bit space (like a large, sturdy box), the moderately sensitive parts a medium 6-bit space (a medium box), and the calm, quiet parts a tight 4-bit space (a small, efficient pouch). This happens without needing to retrain the model or change its brain structure; it just rearranges how the data is stored based on how sensitive each part is to being squished. The researchers found that this approach works beautifully with a new hardware-friendly format called MXINT, which is already supported by modern computer chips.

The results are quite impressive. When tested on massive models like LLaMA-2-70B and LLaMA-3-8B, MXSens managed to keep the model's "voice" clear and accurate while using very little space. Specifically, under a strict W4A4KV4 setting (meaning weights, activations, and key-value caches are all quantized), the method achieved a perplexity score of 3.77 for LLaMA-2-70B and 7.63 for LLaMA-3-8B on the WikiText-2 dataset. Perplexity is a measure of how confused the model is; lower is better. These scores were significantly better than other top methods, proving that by paying attention to the specific sensitivity of different parts of the model, you can get the best of both worlds: a tiny, fast model that still thinks clearly.

The authors suggest that this method is a major step forward because it avoids the heavy overhead of constantly translating numbers, which has slowed down other similar techniques. By using the natural block structure of the new hardware formats, MXSens can mix these different precision levels (4, 6, and 8 bits) seamlessly. The study shows that this sensitivity-guided approach allows for a much better balance between saving space and keeping the model smart, potentially making powerful AI accessible on devices that don't have massive memory banks.

Technical Summary: MXSENS – Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

1. Problem Statement

Large Language Models (LLMs) rely on quantization to reduce memory footprints and accelerate inference, particularly at 4-bit precision. However, aggressive quantization suffers from significant accuracy degradation due to systematic outliers—a small subset of activation values with disproportionately high magnitudes. These outliers distort the distribution within quantization blocks, complicating the selection of scaling factors.

Current solutions generally fall into two categories, both of which have limitations:

Rotation-based methods (e.g., QuaRot, RRS, SpinQuant): These reshape tensor distributions to mitigate outliers. However, they are often incompatible with microscaling formats (like MXINT/MXFP) because rotations introduce group-wise asymmetry that degrades quantization quality. Furthermore, they often leave components like Softmax outputs unquantized.
Mixed-precision methods (e.g., Atom, QoQ, QUIK): These assign higher precision to sensitive channels but typically rely on software-managed FP32 scaling. This necessitates frequent dequantization operations during inference, incurring substantial overhead (up to 90% relative to pure INT4 baselines).

Microscaling formats (e.g., MXINT) offer a hardware-efficient alternative by embedding shared power-of-two scale factors per block (e.g., 32 values), eliminating software scaling overhead. However, they are currently incompatible with rotation-based outlier handling, and existing mixed-precision approaches often ignore the nuanced, non-uniform distribution of quantization sensitivity across different layers and columns.

2. Methodology: MXSens

The authors propose MXSens, a training-free, sensitivity-guided mixed-precision quantization method designed specifically for the microscaling (MXINT) format. The core insight is that quantization sensitivity is not uniform; it varies significantly across layers and columns, and outliers exist on a spectrum from "extreme" (rare, high-magnitude) to "mild" (frequent, moderate deviations).

2.1 Sensitivity Analysis

MXSens employs a two-tiered sensitivity analysis to guide bitwidth allocation:

Column-Wise Sensitivity: The authors define sensitivity based on the diagonal of the empirical Hessian matrix ( $H = X^T X$ ). Columns containing high-magnitude outliers exhibit higher Hessian diagonal values, indicating that quantization noise in these columns causes disproportionate output errors.
Layer-Wise Sensitivity: The authors measure the L2 error at the model's final output when a single layer is quantized in isolation. They observe that errors in earlier layers tend to compound as they propagate through the network, making early layers generally more sensitive, though this varies by architecture (e.g., middle layers in Qwen1.5-7B).

2.2 Triplet Quantization Strategy

Instead of a binary choice between low and high precision, MXSens introduces a triplet quantization scheme using three distinct mantissa bitwidths: 4, 6, and 8 bits.

8-bit Precision: Assigned to the top 32 most sensitive columns in each layer. This aligns with the block size of MXINT (32 values), ensuring that all extreme outliers are captured within a single high-precision block per layer.
6-bit Precision: Assigned to columns with "mild" outliers. This intermediate precision addresses the spectrum of sensitivity between extreme outliers and the bulk of the data, offering better efficiency than a simple 4/8 split.
4-bit Precision: Assigned to the remaining, less sensitive columns.

2.3 Sensitivity-Guided Allocation Algorithm

The method allocates bitwidths under a target average bitwidth budget ( $B$ ) using a global control parameter $R$ .

Sorting: For each linear layer, columns are sorted by their sensitivity scores.
Fixed High-Precision Block: The top 32 columns are always assigned 8-bit mantissas.
Dynamic 6-bit Allocation: The number of 6-bit columns is calculated proportionally to the layer's sensitivity score, scaled by $R$ to meet the global average bitwidth constraint.
Default: All remaining columns are assigned 4-bit mantissas.

This approach leverages the block-wise structure of MXINT, allowing for mixed-mantissa implementation without modifying the model architecture or requiring retraining.

3. Key Contributions

Sensitivity-Guided Triplet Quantization: A novel method assigning 4, 6, and 8-bit precisions based on column- and layer-wise sensitivity, achieving state-of-the-art accuracy without architectural changes or fine-tuning.
MXINT Integration: The adoption of the MXINT numeric format, leveraging its block-wise structure and shared exponent to enable efficient mixed-precision implementation compatible with modern hardware (e.g., NVIDIA Blackwell, Qualcomm Cloud AI 100).
Performance Superiority: Demonstrating that MXSens outperforms existing rotation-based and mixed-precision baselines across a wide range of models and tasks.

4. Experimental Results

The authors evaluated MXSens on various models (LLaMA-2/3, Mistral, Qwen1.5, Gemma-2) across language modeling (WikiText-2) and commonsense reasoning tasks (ARC, BoolQ, MMLU, etc.).

LLaMA-2-70B & LLaMA-3-8B: Under the W4A4KV4 setting, MXSens achieved perplexities of 3.77 and 7.63 on WikiText-2, respectively. This substantially improves upon existing baselines (e.g., QuaRot, Atom, QUIK).
Accuracy vs. Compression: On LLaMA-2-13B at ~4.27 bits, MXSens matched the perplexity of Atom (5.32 vs. 5.31) but achieved significantly higher 0-shot accuracy on Commonsense QA tasks (70.3% vs. 63.5%).
Ablation Studies: Experiments varying the average bitwidth (from ~3.28 to 5.75 bits) showed that sensitivity-guided allocation allows models to approach FP16 quality (within 0.6 accuracy points) at roughly one-third of the bit cost.
Hardware Overhead: The authors estimate that at a 4.03-bit budget, only ~1% of arithmetic operations require higher precision, resulting in an estimated 3% latency overhead, which is considered marginal compared to the accuracy gains.

5. Significance and Claims

The paper claims to establish a new balance between accuracy and resource efficiency for LLM quantization. By moving away from rotation-based methods (which conflict with microscaling hardware) and software-heavy scaling, MXSens offers a hardware-friendly path to high-accuracy 4-bit inference.

The authors emphasize that their work demonstrates the necessity of fine-grained, sensitivity-aware precision allocation. They argue that simply applying uniform 4-bit quantization or coarse mixed-precision strategies is suboptimal. Instead, leveraging the specific block-wise constraints of MXINT to assign 8-bit precision to extreme outliers, 6-bit to mild outliers, and 4-bit to the rest, allows for near-lossless compression without the computational penalties of dequantization or the architectural constraints of rotation matrices.

The work is presented as a step toward practical, high-accuracy deployment of MXINT4 on emerging AI accelerators, proving that training-free methods can effectively mitigate the outlier problem inherent in low-bit LLM inference.