🤖 machine learning

When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation

This paper demonstrates that while RoPE-aligned pairwise rotations are analytically optimal for minimizing a specific pooled variance surrogate under strict commutativity constraints, they fail to improve dynamic 4-bit quantization accuracy compared to full-head Hadamard transforms due to a fundamental misalignment between the surrogate's assumptions and the token-wise statistics used by the quantizer.

Original authors: Shuhan Wang, Yilin Luo, Nan Xu, Chi Wang Cheung

Published 2026-08-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shuhan Wang, Yilin Luo, Nan Xu, Chi Wang Cheung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to fit a giant, fluffy cloud of data into a tiny, rigid box. This is the daily struggle of making massive Artificial Intelligence (AI) models smaller and faster so they can run on regular computers or phones. To do this, scientists use a trick called "quantization," which is like squishing the cloud down to a lower resolution. But here's the catch: the cloud has a few super-bright, super-heavy spots called "outliers." If you squish the whole cloud without care, those heavy spots get crushed, and the AI starts making silly mistakes.

To fix this, researchers have been using a method called "rotation." Imagine the cloud is made of spaghetti strands. If you just squish the whole bundle, the thick strands break. But if you twist the bundle first, you can spread the thickness out evenly, making it easier to squish without breaking anything. A popular way to twist these strands is based on something called "RoPE" (Rotary Position Embedding), which is like a special instruction manual that tells the AI how to understand the order of words in a sentence. This manual naturally splits the spaghetti into pairs of strands that are linked together.

The big question this paper asks is: When we twist the spaghetti to make it fit, should we twist the entire bundle of strands at once, or should we just twist the specific pairs of strands that the instruction manual says are linked? It seems logical that respecting the manual's pairs would be better, right? The authors set out to test this idea, hoping to find a smarter, more efficient way to twist the data.

The Twist That Didn't Work

The researchers, Shuhan Wang, Yilin Luo, and their team, decided to put this "pairwise twisting" idea to the test. They built a mathematical theory showing that if you want to twist the data in a way that perfectly respects the RoPE instruction manual, you are mathematically forced to twist only those specific pairs of strands. They even calculated the perfect angle to twist each pair to minimize the "squishiness" (variance) of the data, creating a local optimum that should, in theory, be the best possible twist for that specific pair.

However, when they actually tried this in the real world using dynamic 4-bit quantization (a specific way of shrinking the data), the results were a bit of a plot twist. Despite their perfect mathematical calculations, the "pairwise-only" method actually made the AI worse at understanding language compared to the old method of twisting the entire bundle at once.

In their experiments with four different AI models (including Llama and Mistral), replacing the full-head twist with their new, mathematically perfect pair-twist increased the model's confusion. The paper measures this confusion using a score called "perplexity." A higher perplexity means the model is more lost. They found that the pairwise method increased perplexity by anywhere from +0.05 to +1.33 points across different models. For example, on the Llama-3.2-1B model, the pairwise method made the model 1.33 points more confused than the standard method. Even when they tried to fix the math by only looking at the data being squished (the "K" stream) to calculate the twist angle, the pairwise method still couldn't catch up to the full-head method.

Why Did the Perfect Math Fail?

The authors realized that the problem wasn't that their math was wrong; it was that their math was solving the wrong problem for the specific tool they were using.

Think of it like this: The researchers designed a key (the pairwise rotation) that fits perfectly into a specific lock (the mathematical variance of the data pairs). But the door they were trying to open (the quantizer) didn't use that lock. The door used a different mechanism entirely: it looked at the range of the data across a much larger group of strands, not just the tiny pairs.

The paper explains that the "pairwise" method only has the power to smooth out a heavy spot by spreading it across two strands. But the "full-head" method can spread that heavy spot across 128 strands (or however many are in the model). It's like trying to flatten a giant rock. If you only have a hammer that can hit two spots at once, you'll leave a bump. If you have a hammer that can hit the whole rock at once, you can flatten it completely. The "mixing support"—the number of strands the twist can affect—was the real hero. As the authors tested twisting larger and larger blocks of strands (from 2 up to 128), they saw that the more strands they could mix, the less confused the model became.

The Takeaway

The main lesson here is that just because a method is mathematically perfect for one specific rule (like respecting the RoPE pairs), it doesn't mean it will work best for the actual job (fitting the data into a tiny box). The authors found that for dynamic quantization, the ability to mix and spread data across the entire group of strands is more important than strictly following the pair-by-pair structure of the instruction manual.

They didn't find a new, better way to twist the data; instead, they found a very clear reason why a seemingly smarter, more structured approach actually failed. The "local" perfection of the pairwise twist couldn't overcome the "global" advantage of the full-head twist. In the end, the paper suggests that when designing these AI tools, we need to make sure our mathematical goals match the actual rules of the quantizer we are using, rather than just assuming that following the structural rules of the AI's architecture will automatically lead to better results.

Technical Summary: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation

Problem Statement
Post-training quantisation (PTQ) of Large Language Models (LLMs) at low bit-widths (e.g., W4A4KV4) is heavily influenced by activation outliers. Rotation-based methods address this by applying orthogonal transforms to redistribute activation energy while preserving the network's unquantised function. A critical design choice in these methods is the "mixing support": whether to mix all channels within an attention head or to restrict mixing to specific structural subspaces.

Rotary Position Embedding (RoPE) partitions each attention head into fixed two-dimensional frequency pairs. This structure raises a specific question: Can a transform that respects this decomposition (pairwise mixing) improve quantisation accuracy compared to full-head mixing? While prior work (e.g., FPTQuant) has utilized per-pair rotations that commute with RoPE, it remains unclear if restricting the mixing support to these pairs, while theoretically optimal for certain local objectives, translates to better end-to-end performance under dynamic quantisation.

Methodology
The authors investigate this question by comparing three online transform configurations applied to queries (Q) and keys (K) after RoPE:

Full-head Hadamard: The baseline method mixing all $d_h$ channels in a head.
Pairwise-only: A head-shared rotation restricted to the two channels of each RoPE frequency pair, commuting with RoPE.
Pairwise + Hadamard: The composition of the pairwise rotation and the full-head Hadamard.

The study focuses on dynamic W4A4KV4 quantisation across four model checkpoints (Llama-3.2-1B, Llama-3.2-3B, Llama-3.1-8B, and Mistral-7B-v0.3).

Theoretical Characterization:
The paper establishes a theoretical converse result: for a single head with distinct RoPE frequencies, the only orthogonal maps that commute with RoPE are independent planar rotations within the frequency pairs. This defines the "RoPE-commuting family."

Optimization Objective:
For the evaluated head-shared parameterisation (where one angle is shared across all heads in a layer for a specific frequency pair), the authors derive a closed-form rotation angle $\phi^*_k$ . This angle minimizes the maximum channel variance under a pooled-covariance, position-averaged surrogate. The implementation is verified to attain this analytic minimum.

Estimation Strategies:
The study evaluates different estimators for the shared angle:

Default: Pools raw Q and K moment accumulators across heads before centring.
K-only: Constructs the covariance from K observations alone (since only K is quantised post-transform).
Stream-balanced: Assigns equal weights to Q and K uncentred moments.

Key Contributions

Exact Local Benchmark: The paper provides a converse characterisation proving that for distinct frequencies, the RoPE-commuting orthogonal family consists exactly of independent pairwise rotations. It further derives the closed-form minimax-variance angle for a pooled-covariance surrogate and verifies its attainment.
Scoped Negative Result: Under dynamic W4A4KV4, the head-shared pairwise configuration yields higher perplexity than full-head Hadamard mixing across all tested checkpoints and context lengths (up to 128K). Even when using the verified surrogate optimum $\phi^*_k$ , the pairwise-only configuration performs worse than the baseline.
Evidence on Proxy Alignment and Mixing Support: The paper identifies two factors explaining the discrepancy between local optimality and quantisation performance:
- Objective Mismatch: The closed-form rotation minimizes a position-averaged second moment (pooled covariance), whereas the dynamic quantiser sets its scale based on a tokenwise group range spanning multiple frequency pairs. Optimising the former does not guarantee control over the latter.
- Mixing Support: Pairwise rotations have a mixing support of $b=2$ , whereas full-head Hadamard has $b=d_h$ . An interpolation experiment varying block-Hadamard support from $b=2$ to $b=d_h$ shows that K range, relative quantisation error, and perplexity degradation decrease monotonically as support increases.

Results

Perplexity: Replacing full-head Hadamard with the head-shared pairwise configuration increases perplexity in every short-context comparison (e.g., +1.33 PPL on Llama-3.2-1B, +0.48 PPL on Llama-3.2-3B). This ordering holds for long contexts (up to 128K).
Composition: Composing the pairwise rotation with the Hadamard (Pairwise + Hadamard) satisfies the $\pm 0.05$ -PPL interval criterion relative to the baseline, indicating the pairwise component does not degrade the full-head mixing when added.
Estimator Impact: Estimating the shared angle from K alone improves the pairwise-only configuration on all checkpoints compared to the default estimator, but it does not eliminate the performance gap relative to full-head mixing.
Support Interpolation: As the mixing support increases from 2 channels (pairwise) to the full head, the K range, relative quantisation error, and perplexity gap all decrease, approaching the full-head baseline performance at $b=d_h$ .

Significance and Claims
The paper concludes that structural alignment alone (commuting with RoPE) is insufficient to ensure a rotation reduces quantisation error. The optimality of a transform for a structured surrogate (pooled covariance) does not necessarily reduce the quantisation error if the surrogate and the transform's mixing support are misaligned with the quantiser's scale-setting statistic (tokenwise group range).

The authors modestly claim that their results separate three distinct concepts:

Optimality for a pooled-covariance surrogate.
Improvement in the quantiser's tokenwise range statistic.
End-to-end model quality (perplexity).

The findings suggest that for dynamic quantisation, broader mixing support (full-head) is more effective than structurally constrained pairwise mixing, primarily because the quantiser's scale-setting mechanism operates on a statistic (tokenwise range) that benefits from the larger support size, which the pairwise transform cannot provide. The paper does not claim to have found a superior method but rather provides a negative result clarifying the limitations of RoPE-aligned pairwise mixing in this specific dynamic quantisation regime.

The Twist That Didn't Work

Why Did the Perfect Math Fail?

The Takeaway

Technical Summary: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation

More like this