🤖 machine learning

TORQ: Two-Level Orthogonal Rotation for MXFP4 Quantization

This paper introduces TORQ, a training-free post-training quantization framework that employs two-level orthogonal rotations to theoretically address structural imbalances in MXFP4 activation quantization, thereby significantly narrowing the accuracy gap between 4-bit inference and full-precision models on large language models.

Original authors: Zukang Xu, Xing Hu, Dawei Yang

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zukang Xu, Xing Hu, Dawei Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to pack a massive, chaotic library of books (the "activations" inside a smart AI) into tiny, uniform shipping boxes (the "MXFP4" format) so they can be shipped quickly and cheaply.

The paper argues that simply shoving these books into the boxes doesn't work well. The books are too messy, and the boxes have a very specific, rigid shape. This causes two main problems, which the authors call TORQ (Two-Level Orthogonal Rotation).

Here is how the paper explains the problem and their solution, using simple analogies:

The Problem: Two Ways the Packing Fails

1. The "One Loud Neighbor" Problem (Inter-block Variance Imbalance)
Imagine your library is divided into groups of 32 books. In the current system, each group gets one single "size label" (a scaling factor) for the whole group.

The Issue: In most groups, the books are small and light. But in a few groups, there is one giant, heavy encyclopedia.
The Result: Because of that one heavy book, the label for the entire group has to be set to "Extra Large." This means all the tiny, light books in that same group are now being forced into a giant box. They get crushed, lost, or turned into zero because the box is too big to hold their small details. A few "loud" groups ruin the precision for everyone else.

2. The "Empty Shelf" Problem (Intra-block Codebook Utilization Imbalance)
The shipping boxes (MXFP4) come with a pre-printed list of specific sizes they can hold (a "codebook"). These sizes are spaced out like rungs on a ladder: small, medium, large, extra-large.

The Issue: Real-world AI data is like a crowd of people where 90% are very short and only 10% are tall.
The Result: Almost all the data falls into the "small" rungs of the ladder. The "medium" and "large" rungs sit completely empty and unused. It's like having a warehouse full of giant crates, but you're only using the tiny ones, while the big ones gather dust. This is a massive waste of space and potential.

The Solution: TORQ (The Great Re-arrangement)

Instead of trying to force the messy books to fit the boxes, the authors propose rotating the books before they get packed. Think of it as shuffling the deck of cards so the heavy cards and light cards are mixed evenly, and the tall people and short people are spread out.

They do this in two steps:

Step 1: The Macro Shuffle (Fixing the "Loud Neighbor")

The Analogy: Imagine you have 100 groups of books. Some groups are heavy, some are light. The authors use a mathematical trick (based on the Schur-Horn theorem) to swap books between groups.
The Goal: They move the "heavy" books from the heavy groups into the light groups, and vice versa.
The Result: Now, every single group has roughly the same total weight. No single group is dominated by one giant book. This allows the "size label" for every group to be set to a perfect, medium size, preserving the details of the small books.

Step 2: The Micro Shuffle (Fixing the "Empty Shelf")

The Analogy: Now that the groups are balanced, look inside one group. The books are still clumped together in the "small" section. The authors rotate the books inside the group.
The Goal: They spin the books so they spread out evenly across the entire ladder of sizes. Instead of 90% of the books being "small," they are now distributed so that some hit the "medium" rungs and some hit the "large" rungs.
The Result: They use every single rung on the ladder. No space is wasted. The "codebook" is fully utilized.

The Outcome

By doing this two-step rotation before the AI is compressed (a process called Post-Training Quantization, which means they don't need to re-teach the AI), they achieve amazing results:

Accuracy: They managed to shrink the AI down to 4-bit precision (tiny boxes) without losing much of its "brain power." In tests, their method (TORQ) was much smarter than previous methods, getting scores close to the full-size, uncompressed AI.
Speed & Size: Because the boxes are smaller, the AI runs faster and takes up less memory on devices like phones or servers.
No Extra Work: The "rotation" is calculated once and then baked into the AI's weights. When the AI runs, it doesn't feel any extra slowdown; the rotation happens automatically as part of the math.

In short: The paper says, "Don't try to squeeze a messy, uneven crowd into a rigid box. Instead, gently shuffle the crowd so everyone is evenly spread out, and then put them in the box. This makes the box work perfectly."

Technical Summary: TORQ – Two-Level Orthogonal Rotation for MXFP4 Quantization

1. Problem Statement

As Large Language Models (LLMs) move toward practical deployment, the Microscaling FP4 (MXFP4) format has emerged as a critical standard for low-bit inference due to its balance of high dynamic range and hardware efficiency. However, directly applying MXFP4 to LLM activation quantization results in severe accuracy degradation. The paper identifies that existing rotation-based quantization methods, designed primarily for integer formats (e.g., INT4), fail to address the unique topological structure of Block Floating Point (BFP) formats like MXFP4.

Through theoretical analysis, the authors reveal that the performance drop stems from two structural imbalances between real-world activation distributions and the MXFP4 block floating-point format:

Extreme Inter-block Variance Imbalance: Under fixed-block microscaling, activation variance is highly uneven. A minority of high-variance blocks dominate the total quantization error. These blocks force the shared scaling factor (exponent) upward to accommodate outliers, which causes the resolution for the majority of small-magnitude activations within the same block to be coarsened excessively, often resulting in zeroing out over 90% of elements.
Intra-block Codebook Utilization Imbalance (Codebook Collapse): Real-world activations often follow a heavy-tailed distribution concentrated near zero, which mismatches the geometric progression codebook of MXFP4 (e2m1). This leads to "codebook collapse," where the vast majority of values fall into the minimum quantization interval, leaving high-capacity codewords for large values idle. Statistics show that ~50% of the codeword range is activated by only ~10% of the data.

2. Methodology: TORQ Framework

To address these challenges, the authors propose TORQ (Two-level Orthogonal Rotation for MXFP4 Quantization), a training-free Post-Training Quantization (PTQ) framework. Instead of passively adjusting quantization parameters, TORQ actively rotates the activation coordinate system to align distribution characteristics with the structural properties of MXFP4. The method decomposes the optimization into two mathematically orthogonal sub-problems:

A. Macro-Equilibrium Rotation (Inter-block)

Objective: To eliminate the dominance of high-variance blocks by redistributing activation energy uniformly across all blocks.
Theoretical Basis: Based on the Schur-Horn theorem and convex optimization theory. The authors prove that under a convex error model, equalizing the variance across blocks minimizes the global Mean Squared Error (MSE).
Mechanism: An orthogonal transformation is constructed to ensure that the diagonal elements of the transformed covariance matrix are uniform (i.e., all block variances become equal). This acts as a "soft outlier suppression" mechanism, preventing high-variance blocks from driving up the shared scaling factors and preserving the precision of small-magnitude elements.
Implementation: An efficient iterative Givens rotation algorithm is used to construct the rotation matrix, ensuring convergence within $O(B^2)$ time.

B. Micro-Alignment Rotation (Intra-block)

Objective: To alleviate "codebook collapse" by reshaping the intra-block activation distribution to utilize the full MXFP4 codebook capacity.
Theoretical Basis: Guided by the principle of maximum entropy. The goal is to make the activation distribution "flow" uniformly into each codeword interval within the normalized FP4 space.
Mechanism: The method defines a Codebook Occupancy Equilibrium Loss, which measures the deviation of empirical codeword usage from a uniform distribution.
Optimization: An alternating coordinate descent strategy is employed:
- S-step: Updates block scaling factors to map maximum activation values to the maximum MXFP4 codeword.
- R-step: Updates the intra-block orthogonal rotation matrix using Givens rotations. A heuristic selects column pairs with high imbalance and statistical complementarity, then searches for the optimal rotation angle via finite enumeration (exploiting the piecewise constant nature of the loss function).

Inference Efficiency

TORQ is designed for minimal overhead. The inverse rotation matrices are fused into the weights of the adjacent linear layers during the offline calibration phase. Consequently, the online inference phase incurs negligible additional computational cost, as the rotation is effectively absorbed into the weight matrix multiplication.

3. Key Contributions

Problem Formalization: The paper is the first to systematically identify and formalize the two structural challenges hindering MXFP4 performance: Inter-block Variance Imbalance and Intra-block Codebook Utilization Imbalance.
Dual-Level Framework: It proposes TORQ, the first dual-level rotation framework explicitly designed for Block Floating Point structures. By integrating macro-variance equilibrium and micro-codebook alignment, it achieves a systematic minimization of quantization error.
Theoretical Foundation: The authors provide rigorous proofs showing that variance equilibrium is approximately optimal under convex error models and that codeword occupancy equilibrium approximates the minimization of quantization error under the MXFP4 geometric structure.
Efficient PTQ Pipeline: A fine-tuning-free pipeline is established that requires only minimal calibration data (e.g., 128 samples) and demonstrates hardware friendliness with negligible parameter footprint.

4. Experimental Results

Experiments were conducted on mainstream LLMs (LLaMA3 series and Qwen3 series) ranging from 8B to 70B parameters, covering both Dense and Mixture-of-Experts (MoE) architectures. Evaluation included perplexity on WikiText2 and zero-shot accuracy on tasks such as ARC, HellaSwag, MMLU, GSM8K, and HumanEval.

Performance Superiority: TORQ significantly outperforms existing State-of-the-Art (SOTA) methods (including RTN, GPTQ, QuaRot, OSTQuant, FlatQuant, and BRQ).
- On Qwen3-32B, TORQ reduced WikiText perplexity to 8.43 (compared to 274.93 for RTN and 14.62 for OSTQuant), approaching the BF16 baseline of 7.61.
- Average accuracy on zero-shot tasks for Qwen3-32B increased from 38.40% (RTN) to 73.63% (TORQ), nearly matching the BF16 baseline of 74.82%.
- On LLaMA3-70B, TORQ achieved a perplexity of 3.49 (vs. 3180.43 for RTN) and an average accuracy of 76.28%.
Reasoning Capabilities: TORQ demonstrated strong recovery of "deep intelligence" in complex reasoning tasks. On GSM8K for Qwen3-8B, accuracy was restored to 86.82% (vs. 67.37% for GPTQ), and HumanEval accuracy reached 61.15%.
Generalizability: The method proved effective across different microscaling formats (MXINT4, NVFP4) and MoE architectures (Qwen3-30B-A3B), confirming its format-agnostic nature.
Efficiency: TORQ calibration is significantly faster than competing methods (e.g., ~10x faster than FlatQuant for Qwen3-32B). Inference speedups of 3.5x over BF16 were observed on Qwen3-30B-A3B, with memory usage reduced from 56.9GB to 15.3GB.

5. Significance and Claims

The paper claims that TORQ effectively bridges the gap between 4-bit floating-point quantization and full-precision inference, solving the accuracy degradation inherent to MXFP4 activation quantization. By shifting the optimization paradigm from local parameter adjustment to "structure-aware orthogonal transformation," TORQ provides the first systematic solution for MXFP4 activation quantization.

The authors position this work as laying a crucial foundation for the application of low-bit floating-point formats in the efficient deployment of LLMs. They acknowledge limitations, such as the current focus on first-order statistics (ignoring higher-order moments like kurtosis) and the use of fixed block sizes, noting that adaptive strategies for these factors remain a direction for future work. However, the current results establish a new optimization paradigm for low-bit floating-point activation quantization.