🤖 machine learning

WUSH: Near-Optimal Adaptive Transforms for LLM Quantization

The paper introduces WUSH, a near-optimal adaptive transform that combines a Hadamard backbone with data-dependent second-moment adjustments to significantly improve low-bit LLM quantization accuracy and throughput compared to existing fixed methods.

Original authors: Jiale Chen, Vage Egiazarian, Roberto L. Castro, Torsten Hoefler, Dan Alistarh

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiale Chen, Vage Egiazarian, Roberto L. Castro, Torsten Hoefler, Dan Alistarh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to pack a massive, delicate library of books (a Large Language Model) into a tiny, rugged suitcase (a low-bit quantized format) so you can carry it anywhere. The goal is to make the suitcase small and light without crushing the books inside.

The problem is that a few books in this library are extremely heavy and oddly shaped (these are called "outliers"). If you try to pack them using a standard measuring tape, these heavy books force the whole suitcase to be sized for them, leaving the rest of the books with very little room. This results in a messy, inefficient packing job where the smaller books get squished and lose their details.

The Old Solution: The "One-Size-Fits-All" Rotation

Previously, researchers tried to fix this by rotating the books before packing them. Imagine taking all the books and spinning them 45 degrees (a Hadamard rotation). This spreads the weight of the heavy books out more evenly across the suitcase, so no single corner gets crushed.

However, this rotation was fixed. It didn't matter if the heavy books were on the left, right, top, or bottom; the rotation was always the same. It was a "data-agnostic" approach—blind to the specific contents of the suitcase. While it helped, it wasn't the perfect solution.

The New Solution: WUSH (The Smart, Custom Tailor)

The paper introduces WUSH, a new method that acts like a smart, custom tailor for your suitcase. Instead of using a fixed rotation, WUSH looks at the specific books you are packing right now and calculates the perfect way to rearrange them.

Here is how WUSH works, broken down simply:

It Takes a Snapshot: WUSH looks at the specific "weights" (the books) and "activations" (the way the books are being used) to understand exactly where the heavy outliers are.
It Builds a Custom Map: Using a mathematical formula (closed-form), it creates a unique transformation map for every small group of books. It combines two things:
- The Standard Spin: It keeps the reliable 45-degree rotation (Hadamard) that everyone knows works well.
- The Custom Adjustment: It adds a second, data-specific layer that fine-tunes the arrangement based on the actual weight distribution of this specific group of books.
The Result: This creates a non-orthogonal (non-perfectly square) shape that perfectly balances the heavy and light books. It's "near-optimal," meaning it gets as close to the theoretical perfect packing as math allows.

Why It's a Big Deal

Better Accuracy: When the authors tested this on models like Llama-3.1 and Qwen, WUSH packed the books much better than the old fixed rotations. For example, on a specific test, it improved the model's "intelligence" (accuracy) by nearly 3 points compared to the best previous method. That's a huge jump in the world of AI.
It's Fast: You might think calculating a custom map for every group of books would be slow. But the authors built a special GPU engine (a high-speed kitchen) that does this custom packing instantly. It's so fast that it runs almost as quickly as the old, simpler method, delivering speeds up to 5.8 times faster than the standard high-precision format (BF16).
It Works with New Formats: The paper shows WUSH works great with new, experimental formats (like MXFP4) that are designed to be super efficient but were previously hard to use because of those heavy outliers.

The Bottom Line

Think of WUSH as upgrading from a generic, pre-made suitcase to a 3D-printed, custom-fit case that molds perfectly around your specific luggage. It doesn't just rotate the items; it reshapes the space around them to eliminate waste and prevent crushing.

The paper claims that this method is mathematically proven to be nearly the best possible way to handle these outliers, and in real-world tests, it makes AI models smaller, faster, and smarter without needing extra computing power.

Technical Summary: WUSH: Near-Optimal Adaptive Transforms for LLM Quantization

1. Problem Statement

Quantizing Large Language Model (LLM) weights and activations is a standard approach for efficient deployment. However, a central difficulty arises from "outliers"—a few extreme values in weights and activations that stretch the dynamic range. These outliers force quantization scales to be large, thereby degrading the effective resolution of low-bit representations and amplifying quantization errors.

Prior mitigation strategies often employ linear transforms (e.g., rotations) before quantization to spread variance more evenly across channels. While Hadamard rotations have proven empirically effective, they are typically fixed and data-agnostic. They do not adapt to the specific statistics of the underlying weights or activations. This raises a fundamental question: if the Hadamard transform is not data-aware, in what sense can it be considered optimal for quantization? Existing methods that learn transforms (e.g., SpinQuant, FlatQuant) rely on iterative optimization (gradient descent) on calibration data, which can be costly and heuristic.

2. Methodology

The authors propose WUSH, a closed-form, data-aware linear blockwise transform designed for joint weight-activation quantization. The method is derived to minimize the $L_2$ output loss under standard Round-to-Nearest (RTN) AbsMax-scaled block quantizers, covering both floating-point (FP) and integer (INT) formats.

Theoretical Derivation

The problem is formulated as minimizing the expected quantization error for a block of weights $W$ and activations $X$ . The authors model the quantization error probabilistically:

For Floating-Point (FP): The error is modeled as proportional to the absolute value of the input ( $\varepsilon(\alpha) = \text{diag}(\eta)\alpha$ ).
For Integer (INT): The error is modeled as proportional to the maximum absolute value within a group ( $\varepsilon(\alpha) = \|\alpha\|_\infty \eta$ ).

By analyzing the second-order statistics (covariance) of the weight and activation blocks, the authors derive a closed-form optimal transform. Let $W'$ and $X'$ be matrices derived from the Cholesky decomposition of the normalized second moments of weights and activations, respectively. Let $U, S, V$ be the Singular Value Decomposition (SVD) of $W'^\top X'$ . The optimal transforms $T_W$ and $T_X$ are constructed as:
$T_X = H S^{-1/2} V^\top X'^\top$
$T_W = H S^{-1/2} U^\top W'^\top$
where $H$ is a normalized Hadamard matrix.

Key properties of this construction:

Data-Aware: Unlike fixed Hadamard transforms, WUSH incorporates $S^{-1/2}$ and the SVD components ( $U, V$ ), which depend on the specific second-order statistics of the data.
Non-Orthogonal: The resulting transform is generally non-orthogonal, allowing it to balance scales between weights and activations in a way orthogonal transforms cannot.
Optimality: The paper proves that this construction is provably optimal for FP block quantizers and asymptotically optimal for INT block quantizers (within a factor of $d$ or $d^{o(1)}$ depending on the distribution).

Integration with GPTQ

The method naturally integrates with GPTQ (Greedy Per-Block Quantization). Since GPTQ utilizes the Hessian (second-order information) for error propagation, WUSH can utilize the same statistics to construct the transform. The authors propose an interleaved computational schedule where weight updates and transform construction are coupled, allowing WUSH to be applied alongside GPTQ with negligible additional overhead.

Implementation

To address the challenge of applying a distinct transform per block during inference, the authors implement a fused GPU kernel. The WUSH transform matrices are stored in a transposed layout $(G, G, C)$ to maximize memory bandwidth utilization. The kernel fuses the blockwise transform with activation quantization, achieving throughput comparable to optimized Hadamard kernels.

3. Key Contributions

Closed-Form Optimal Transforms: The derivation of the first closed-form, data-aware linear transforms for joint weight-activation quantization that are provably near-optimal for both FP and INT formats.
Theoretical Insight: The paper clarifies the role of the Hadamard transform. It shows that while Hadamard is an effective data-agnostic orthogonal transform, the optimal solution requires a non-orthogonal component ( $S^{-1/2}$ ) derived from data statistics to truly minimize quantization error.
Efficient Implementation: A specialized fused GPU kernel that handles per-block adaptive transforms without sacrificing inference speed, matching the performance of fixed-transform baselines.
Empirical Validation: Extensive experiments demonstrating significant accuracy improvements over state-of-the-art baselines (Hadamard-based MR-GPTQ, SmoothQuant, QuaRot, SpinQuant) across multiple model families (Llama, Qwen) and quantization formats (MXFP4, NVFP4, INT4).

4. Experimental Results

The paper evaluates WUSH on Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, and Qwen3 models (8B, 14B, 32B) using the LM Evaluation Harness and Platinum Benchmarks.

Layerwise Loss: WUSH consistently achieves the lowest layerwise RTN quantization loss across MXFP4, NVFP4, and INT4 formats compared to Identity, Random Rotation, Hadamard, and non-adaptive variants.
End-to-End Accuracy:
- On Llama-3.1-8B-Instruct with MXFP4 (W4A4), WUSH improves accuracy by +2.8 average points over the Hadamard-based baseline (RTN) and +0.7 points (GPTQ).
- It substantially narrows the performance gap between NVFP and MXFP formats. For example, on Qwen3-14B, WUSH reduces the NVFP4-MXFP4 average gap to 0.36 points.
- WUSH combined with GPTQ achieves the best results across most benchmarks, often recovering over 96% of the BF16 performance in W4A4 settings.
Throughput: The fused kernel implementation delivers up to 5.8× per-layer throughput over BF16 on RTX 5090 GPUs, with only a 1.3% average throughput difference compared to the optimized Hadamard baseline.
Overhead: Offline preprocessing costs are comparable to standard GPTQ. Storage overhead for the activation-side transforms is negligible (e.g., ~1.4% for MXFP4 on Qwen3-8B).

5. Significance and Claims

The paper claims that WUSH clarifies the empirical success of Hadamard rotations by showing they are a specific, data-agnostic component of a broader optimal solution. By combining a fixed Hadamard backbone with a data-dependent second-moment component, WUSH achieves a provably near-optimal non-orthogonal transform.

The authors emphasize that their approach is principled and data-aware, moving beyond heuristic or iterative learning methods. They state that WUSH makes the MXFP format competitive with NVFP, addressing a key limitation in current low-precision inference. The work demonstrates that principled, data-aware block transforms can yield significant improvements in accuracy without compromising the efficiency required for large-scale LLM deployment.