🤖 machine learning

Influence-Inspired Spectral Rotations for Extreme Low-Bit LLM Quantization

This paper introduces "BBT-spectral," an engineering-focused quantization method that applies influence-adaptive Walsh-Hadamard rotations and energy-based rescaling to weight matrices, significantly reducing perplexity in extreme low-bit (W2A16) LLM quantization across various model architectures while ensuring hardware compatibility with Intel devices.

Original authors: Gorgi Pavlov

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gorgi Pavlov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly detailed library of knowledge (a Large Language Model) that you want to shrink down to fit into a tiny, cheap backpack (extreme low-bit quantization). The problem is that when you try to squash this library, you inevitably lose some pages or blur the text, making the model confused and less accurate.

This paper introduces a clever trick called BBT-spectral to solve this. Instead of just randomly squashing the data, it rearranges the books before packing them so that the most important information gets the best protection.

Here is how it works, using simple analogies:

1. The Problem: The "One-Size-Fits-All" Box

Normally, when we compress these models, we treat every part of the data equally. Imagine you have a box with 64 slots. You put 64 different items in there and try to fit them all into a smaller space. If you just shrink everything by the same amount, the delicate, fragile items (the most important "spectral" signals) get crushed, while the sturdy items (less important noise) take up too much space. The result is a messy, confused model.

2. The Solution: The "Spectral Shuffle"

The authors propose a two-step dance before the compression happens:

Step A: The Magic Shuffle (Walsh-Hadamard Rotation):
Think of the model's data as a deck of cards. The authors use a specific, fixed mathematical shuffle (called a Walsh-Hadamard transform) to mix the cards. This doesn't change the total information, but it rearranges it so that the "loud" and "important" signals group together in specific columns, while the "quiet" noise moves to others. It's like sorting a messy pile of laundry so all the expensive silk shirts are in one pile and the old socks are in another.
Step B: The Custom Sizing (Spectral Scaling):
Now that the important signals are grouped together, the authors apply a "volume knob" to each column. They turn up the volume on the columns holding the important "silk shirts" (high energy) and turn down the volume on the "socks" (low energy).
- Why? When the model is finally squashed (quantized) into tiny 2-bit or 4-bit numbers, the "loud" columns get a larger, more precise grid to land on. The "quiet" columns get a smaller, rougher grid.
- The Result: The compression algorithm makes fewer mistakes on the important parts because it has been given more "room" to be accurate there.

3. The "No-Loss" Guarantee

The authors emphasize that this rearranging and resizing is mathematically perfect before the compression happens. If you were to reverse the process, you would get the exact original model back, down to the last decimal point. It's like rearranging furniture in a room; the room looks different, but the furniture is exactly the same until you actually start packing it into boxes.

4. Handling Tricky Architectures (The "Special Cases")

The paper admits that this "Magic Shuffle" didn't work perfectly for every type of model architecture right away. Some models had special "gates" or "norms" that got confused by the shuffle. The authors built three specific "patches" to fix these:

The "Head" Fix: For some models, they had to rotate the data inside the attention heads (like adjusting the lenses on a pair of glasses) to keep the math working.
The "Pair" Fix: For other models, they rotated data in pairs to ensure it didn't clash with the model's internal clock (RoPE).
The "Gate" Fix: They found a bug where a specific type of "gate" in the model wasn't being scaled correctly, and they fixed the code to include it.

5. The Results: Big Wins on Tiny Models

The authors tested this on several models (ranging from small to medium-sized).

The Outcome: When they compressed these models down to just 2 bits (extremely small), the new method made the models 15% to 58% more accurate (measured by how well they predict the next word) compared to the standard method.
The Catch: The more the model struggled with the standard method, the bigger the improvement. It's like a lifeboat that saves the most people when the ship is sinking the fastest.
The Limit: When they tried this on slightly larger models (4 bits), the improvement disappeared. This makes sense: if you have a big enough box (4 bits), you don't need to be as clever about rearranging the furniture. The trick is specifically for when the box is very small.

Summary

In short, this paper says: "Don't just crush your AI model into a small space. First, shuffle the data so the important parts are easy to spot, give those parts extra protection, and then crush it. This makes the tiny models much smarter without needing to train them from scratch or use complex, slow learning algorithms."

The authors are careful to say this is an engineering trick that works great in practice, even if the deep mathematical theory behind why it works (connecting to Boolean logic) is still being explored. They proved it works on real hardware and didn't break the math.

Technical Summary: Influence-Inspired Spectral Rotations for Extreme Low-Bit LLM Quantization

Problem Statement

The paper addresses the challenge of extreme low-bit weight-only quantization (specifically W2A16, 2-bit weights, 16-bit activations) for Large Language Models (LLMs). At such low bit-widths, standard quantization methods (like vanilla Intel Auto-Round) suffer from significant reconstruction errors, leading to drastic increases in perplexity (PPL) on benchmarks like WikiText-2. The core difficulty lies in the "quantization noise budget": with only 2 bits, the rounding error is large, and standard per-group quantization fails to prioritize the most critical channels, causing severe degradation in model performance, particularly in models with wide MLP layers or specific architectural constraints (e.g., Qwen3-style attention).

Methodology

The proposed method, BBT-spectral, applies a fixed, mathematically invariant pre-quantization transformation to linear layers before handing them off to a standard reconstruction-error quantizer (Intel Auto-Round). The approach is inspired by a companion theory paper on Boolean influence and Walsh-Hadamard geometry but is applied heuristically to real-valued LLM layers.

1. Core Transformation

The method consists of two steps applied to a linear layer's weight matrix $W$ and input $x$ :

Spectral Rotation: The input is rotated into the Walsh-Hadamard basis using a fixed orthogonal matrix $Q = H_{dpad}/\sqrt{dpad}$ , where $H$ is the Walsh-Hadamard matrix and $dpad$ is the zero-padded dimension to the next power of two.
Per-Coordinate Scaling: The columns of the rotated weight matrix are scaled by a diagonal matrix $D = \text{diag}(s)$ $D = diag (s)$ , where the scale factors $s_\ell$ $s_{ℓ}$ are derived from the spectral activation energy of the input coordinates.
- The spectral energy proxy $\rho_\ell$ is calculated as the expected squared magnitude of the $\ell$ -th coordinate after WHT rotation: $\rho_\ell = E[h_\ell^2] / \sum E[h_k^2]$ .
- The scaling factor is defined as $s_\ell = (1 + d_{pad} \cdot \rho_\ell)^\alpha$ , with $\alpha = 0.5$ .

Mathematical Invariance: The transformation is an exact identity at full precision ( $x W^T = (x Q D^{-1}) (D Q^T W^T)$ ). The scaling "boosts" high-energy channels before quantization and divides them back after dequantization, effectively biasing the integer rounding budget toward channels that carry more spectral signal.

2. Architectural Extensions

The basic recipe initially failed on models with specific attention mechanisms. The paper introduces three extensions to handle these cases:

Spectral-PCA (for Qwen3-style): For architectures with per-head $q\_norm/k\_norm$ layers (e.g., Qwen3), the standard input-side scaling breaks the norm's denominator. The authors replace the norm with a Matrix-Γ RMSNorm that rotates the projection weights via per-head PCA ( $U_h$ ) and applies a diagonal matrix $\Gamma_h$ to restore the vanilla output exactly.
Pair-PCA (for RoPE-commuting architectures): For models without $q\_norm$ (e.g., Llama, SmolLM, Qwen2.5), the rotation must commute with Rotary Position Embeddings (RoPE). Since RoPE acts as 2D rotations, the method computes a constrained SO(2) PCA for each RoPE pair, rotating the weight rows in 2D planes to align with activation variance without breaking RoPE invariance.
MoE-Aware Absorption: A fix identified via "architectural fuzzing" on a synthetic Laguna model, extending the input-side absorption list to include $g\_proj$ (head gates) in addition to standard $q/k/v\_proj$ to prevent non-linearity leakage.

Key Contributions

Drop-in Spectral Transformation: A principled, non-learned pre-quantization step (WHT rotation + energy-based scaling) that improves W2A16 quantization without requiring gradient-based rotation learning.
Empirical Validation: Demonstrated 15–58% reduction in WikiText-2 PPL across four models (SmolLM-135M/360M, Qwen2.5-0.5B/1.5B) compared to vanilla Auto-Round. The largest gains were observed in models with the most degraded vanilla baselines (e.g., Qwen2.5-0.5B saw a 57.9% improvement).
Architectural Generalization: Successfully extended the method to complex architectures (Qwen3 with $q\_norm$ , RoPE-heavy models) via Spectral-PCA and Pair-PCA, resolving previous failure modes.
Bit-Width Ablation: Showed that the method's payoff is Schur-convexity consistent: the benefit scales with the quantization noise budget. The method provides massive gains at W2 but yields negligible (noise-floor) improvements at W4, confirming it targets the high-noise regime.
Engineering Robustness: Validated that the method introduces no extra runtime operators (all transforms are folded into weights) and maintains device-invariant PPL across Intel NPU, GPU, and CPU.

Results

Performance:
- SmolLM-135M: PPL dropped from 81.03 to 45.11 (−44.3%).
- Qwen2.5-0.5B: PPL dropped from 119.31 to 50.22 (−57.9%).
- Qwen2.5-1.5B: PPL dropped from 36.93 to 28.16 (−23.7%).
- Qwen3-0.6B: Using Spectral-PCA, PPL dropped from 136.76 (vanilla) to 88.99 (35% improvement over vanilla).
Throughput: The folded weights result in standard dense transformer graphs. Throughput is comparable to vanilla baselines (within ±15%), with no additional runtime overhead.
Noise Floor: At W4, the improvement was +0.06 PPL (statistically insignificant), confirming the method is specific to extreme low-bit regimes.

Claims and Limitations (Author's Voice)

The authors explicitly frame this as an applied engineering paper rather than a theoretical breakthrough:

No Formal Equivalence: They do not claim the real-valued WHT activation energy used here is formally equivalent to the Boolean influence defined in the companion theory paper. The connection is intuitive and qualitative (per-coordinate spectral mass), not a formal theorem transfer.
No Statistical Significance: Results are single-seed. Differences below ±0.5 PPL are treated as evaluation noise.
Baseline Scope: Comparisons are made against vanilla Auto-Round. The paper does not claim superiority over other strong low-bit baselines like AWQ, GPTQ, SpinQuant, or QuaRot; head-to-head comparisons are deferred to future work.
Scope: The method is a "drop-in" heuristic. It does not claim to solve all quantization issues, specifically noting that the "Schur-convexity" argument is a heuristic guide for the real-valued setting, not a proven theorem for it.

The primary contribution is a small, principled, exactly math-invariant transformation that yields non-trivial empirical wins at W2 across diverse model families, offering a practical solution for extreme low-bit deployment where standard rounding fails.