💬 NLP

ARCHead: Activation-Metric Residual Correction for Large Language Model Output Heads

ARCHead is a novel activation-metric residual correction method that compresses large language model output heads using a quantized low-rank core and group-wise INT4 residuals, achieving significant storage reduction (3.7–3.9x) while maintaining near-lossless perplexity and minimal throughput impact.

Original authors: Şuayp Talha Kocabay, Talha Rüzgar Akkuş, Kamer Ali Yuksel

Published 2026-08-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Şuayp Talha Kocabay, Talha Rüzgar Akkuş, Kamer Ali Yuksel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to fit a massive, intricate library into a tiny backpack. This is the daily challenge for engineers working with Large Language Models (LLMs), the super-smart AI brains behind chatbots and creative writers. These models are built from billions of tiny mathematical switches called "weights." To make them fast and cheap to run on regular computers, scientists use a trick called quantization. Think of quantization as translating a high-definition, 4K movie into a lower-resolution, 1080p version. You lose a little bit of picture quality, but the file size shrinks dramatically, making it easy to carry around.

However, there's a catch. While engineers have gotten very good at compressing the main "thinking" parts of the AI (the transformer blocks), they often leave the very last part—the part that actually chooses the next word to say—uncompressed. It's like packing your backpack with lightweight, compressed clothes but forgetting to compress your heavy, bulky winter coat. This "coat" is called the LM-head (Language Modeling head). It's a giant, dense map that connects the AI's thoughts to every possible word in its vocabulary. Because it's so big and left in its original, heavy format, it takes up a huge amount of space, sometimes even more than all the compressed clothes combined. If you want to truly shrink these AI models, you have to figure out how to fold this heavy coat without tearing it.

This is where a new method called ARCHead comes in. The researchers behind it realized that simply squashing this heavy coat with a standard "low-bit" compression (like turning everything into tiny 4-bit numbers) was too blunt an instrument. It was like trying to fit a delicate, complex sculpture into a box by just smashing it; the result was a distorted mess that made the AI stutter and make silly mistakes.

Instead, ARCHead uses a clever, two-step strategy that acts more like a master tailor than a sledgehammer. First, it takes the heavy coat and creates a "skeleton" version of it—a low-resolution sketch that captures the general shape using very few bits. But a sketch isn't perfect. So, the second step is the magic: ARCHead adds a "correction layer." This isn't just random noise; it's a smart, low-rank patch that specifically fixes the errors in the places where the AI's brain is most active. Imagine you have a rough sketch of a face, and you know exactly which features (like the eyes or smile) are most important for recognizing the person. ARCHead adds tiny, precise details only to those important features, ignoring the less critical background.

The results are impressive. When the team tested this on a model called Qwen3-8B-Base, they found that ARCHead could shrink the storage needed for this "coat" by nearly 4 times (specifically, it used only about 25.6% of the space the original heavy version took). Even better, the AI's performance barely dipped. The "perplexity" (a score measuring how confused the AI is) stayed incredibly close to the original, with a relative score of 1.007, compared to a much worse 1.15 for standard compression methods.

The paper also shows that this method works like a perfect "drop-in" replacement. If you already have a compressed AI model using other popular tools like AWQ or bitsandbytes, you can swap out their heavy, uncompressed coat for an ARCHead one without breaking anything. It adds a tiny amount of extra confusion (about 0.006 to 0.007 in cross-entropy), which is barely noticeable, but saves a massive amount of space. The researchers measured that this didn't slow down the AI's speed; the generation rate stayed almost exactly the same, changing by less than 2%.

In short, ARCHead doesn't try to reinvent the wheel or compress the whole AI from scratch. Instead, it solves the specific, annoying problem of the "leftover" heavy coat that other methods ignore. By using a smart, activity-aware correction system, it proves that you can make these giant AI models much smaller and easier to carry without sacrificing the quality of their conversation. It's a practical, measured step forward, suggesting that with the right folding technique, even the bulkiest parts of our digital brains can fit into a pocket.

Technical Summary: ARCHead

Problem Statement

While post-training quantization (PTQ) methods like GPTQ, AWQ, and bitsandbytes have successfully compressed the transformer blocks of Large Language Models (LLMs) to 4-bit or lower, practical deployment pipelines often retain the final language-modeling head (LM-head) in full precision (BF16 or FP16). This head, which maps hidden states to vocabulary logits, can constitute a massive dense tensor. For instance, in Qwen3-8B-Base, the BF16 output projection occupies approximately 1.18 GB, becoming one of the largest remaining dense parameters after block quantization.

Standard low-bit quantization of this head fails to preserve quality because it minimizes weight-space reconstruction error (Frobenius norm) without accounting for the anisotropic distribution of hidden states. A small error in a frequently activated direction can significantly perturb many vocabulary logits, whereas a larger error in an inactive direction may have negligible impact. Consequently, standard INT4 quantization of the head leads to substantial perplexity degradation (e.g., a relative PPL of 1.151 on Qwen3-8B-Base compared to 1.000 for the dense head).

Methodology: ARCHead

The authors propose ARCHead (Activation-Metric Residual Correction), a specialized, packed LM-head compressor designed to replace the dense BF16 head without retaining a dense copy. The method decomposes the head into three components:

Quantized Low-Rank Core: The head is first approximated by a low-rank factorization ( $A_c B_c$ ) and a group-wise signed-INT4 residual ( $Q_{SC4}$ ). This forms a compact core representation.
Activation-Metric Transform: Unlike standard methods that treat all dimensions equally, ARCHead estimates the activation geometry using calibration data ( $H$ ). It constructs a damped covariance matrix $C_\lambda = H^\top H / N + \lambda \bar{c} I$ and derives a metric transform $T_p = Q \Lambda^p Q^\top$ . This transform reweights the error space to emphasize directions that are frequently activated.
Low-Rank Residual Correction: The residual error left by the quantized core ( $E = W - W_d$ ) is transformed by $T_p$ . A rank- $r_r$ truncated SVD is performed on this transformed residual to find the optimal low-rank correction ( $A_w B_w$ ) in the activation-derived metric. The factors are then quantized (INT8) and packed.

Theoretical Basis: The paper proves that for a fixed quantized core and an invertible metric transform, the derived low-rank correction is the best rank- $r$ approximation in the induced activation-metric space (a damped form of the empirical logit Mean Squared Error). This optimality holds before the final factor quantization.

Implementation: The final module stores only the packed quantized core, the INT4 residual with scales, and the INT8 correction factors. The original dense BF16 matrix is discarded after construction. The inference computation is factorized: $\hat{Y} = H W_d^\top + (H B_w^\top) A_w^\top$ , avoiding the reconstruction of a dense matrix.

Key Contributions

Activation-Metric Formulation: The authors formulate output-head compression using an activation-derived metric, demonstrating that the ARCHead correction provides the optimal rank- $r$ approximation for a fixed core in this specific metric.
Packed Representation: They develop a state-dictionary footprint that is 25–27% of the corresponding BF16 head across five model families, achieving a persistent storage reduction of 3.7–3.9×.
Quality Preservation: ARCHead avoids the significant quality loss associated with storage-matched standard INT4 quantization. It generalizes across different architectures and can compress the dense heads left by existing block quantizers (AWQ, bitsandbytes) with minimal additional loss.
Comprehensive Evaluation: The work evaluates logit fidelity, downstream accuracy, calibration sensitivity, construction time, and generation throughput, explicitly separating persistent parameter storage from backend-dependent runtime peak memory.

Experimental Results

The method was evaluated on models including Qwen3-8B-Base, Gemma-4-E4B, and VibeThinker-3B.

Storage Efficiency: On Qwen3-8B-Base, ARCHead reduces the head storage from 1187 MB (BF16) to 304 MB (a 3.9× compression).
Perplexity: On Qwen3-8B-Base, ARCHead achieves a relative perplexity of 1.007, nearly matching the dense BF16 head (1.000). In contrast, storage-matched standard Group INT4 yields 1.151, and SVD8+INT4 yields 1.211.
Hybrid Deployment: When replacing the BF16 head left by AWQ or bitsandbytes NF4, ARCHead adds only 0.006–0.007 to the cross-entropy loss, with less than a 2% change in throughput.
Comparison to GPTQ-Style Head: Compared to a GPTQ-style INT4 head quantizer using the same calibration budget, ARCHead achieves lower mean relative perplexity and logit MSE, reducing GPTQ's degradation by 33–49% across various token budgets. It also constructs the head 2.36–2.58× faster.
Fidelity: ARCHead preserves the dense top-1 token on 93.05% of positions, significantly outperforming the ~76% agreement of standard INT4 baselines.
Throughput: Generation throughput changes by less than 2% in all measured configurations, indicating no significant inference bottleneck.

Significance and Claims

The paper positions ARCHead as a complementary solution to existing block quantizers rather than a replacement. Its primary significance lies in addressing a concrete gap in LLM deployment: the large, dense, and often unquantized output projection that remains after transformer blocks are compressed.

The authors claim that ARCHead provides a practical quality-storage trade-off, enabling the compression of the final projection layer to a similar storage footprint as standard INT4 but with quality metrics nearly indistinguishable from the full-precision head. They emphasize that their memory claims refer specifically to persistent parameter storage (serialized state dictionary) and load-time memory, distinct from runtime peak memory which includes activations and KV caches.

The work is modest in its scope, acknowledging that:

It is specialized for output heads and does not compress MLP or attention weights.
The benefit is most pronounced when the vocabulary projection is dense and sensitive to standard low-bit quantization (e.g., Qwen), whereas models like Gemma-4-E4B already show strong performance with standard INT4.
The "conditional optimality" applies to the unquantized correction for a fixed core; the end-to-end quantized module is an approximation of this ideal.
Downstream evaluations are limited to sanity checks (HellaSwag, TruthfulQA, WinoGrande) and do not claim to preserve every model capability or safety property.