🤖 machine learning

MARR: Module-Adaptive Residual Reconstruction for Low-Bit Post-Training Quantization

This paper proposes MARR, a module-adaptive residual reconstruction method for low-bit post-training quantization that employs a PID-based strategy to dynamically assign module-specific scaling coefficients, effectively balancing accumulated-error correction and Hessian-approximation bias to significantly improve performance in LLMs and ViTs.

Original authors: Le Su, Xing Luo, Zhi Jin

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Le Su, Xing Luo, Zhi Jin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Shrinking a Giant Without Losing Its Brain

Imagine you have a massive, incredibly smart library (a Large Language Model or AI) that takes up an entire city block. You want to shrink it down to fit into a small backpack so you can carry it around easily. This process is called Quantization.

To make the library smaller, you have to rewrite all the books using fewer letters (lower precision). Usually, you try to keep the books at 8 letters per word. But to save even more space, you want to shrink them to just 2 or 4 letters per word (Low-Bit Quantization).

The problem? When you shrink the words too much, you lose details. The library starts making mistakes. If you shrink one book, the next book might get confused because it was expecting the original, detailed version. These small mistakes pile up, like a game of "telephone," until the story makes no sense by the end.

The Previous Solution: The "Residual" Fix

Researchers previously tried to fix this by adding a "correction note" (called a Residual) between the books.

How it worked: If Book A gets shrunk and loses a detail, the correction note says, "Hey, Book B, remember that missing detail from Book A."
The Result: This helped a lot. It stopped the mistakes from piling up.

The New Problem: The "Over-Correction" Bias

The authors of this paper discovered a catch. While these correction notes help, they can also be too strong.

Imagine a mechanic trying to fix a car engine.

The Problem: The mechanic (the AI) assumes the engine is perfectly smooth (a mathematical assumption called Hessian-Approximation).
The Side Effect: When the mechanic adds a huge "correction note" to fix a small rattle, they might accidentally introduce a new, bigger vibration because their assumption about the engine wasn't 100% perfect.
The Analogy: It's like trying to balance a scale. If you add too much weight to one side to fix a tilt, you might tip the scale the other way because you didn't account for the exact shape of the weight you added.

In technical terms, the "correction notes" (residuals) introduce a new kind of error called HA Bias. If the correction is too strong, the AI gets confused. If it's too weak, the original mistakes pile up.

The Solution: MARR (The "Smart Thermostat" for AI)

The authors propose a new method called MARR (Module-Adaptive Residual Reconstruction).

1. The "One-Size-Fits-All" Doesn't Work

Previously, researchers used one single rule for the whole library: "Add a correction note of strength 1.0 to every book."

The Flaw: Some books are delicate; others are sturdy. A correction that helps a sturdy book might ruin a delicate one. The paper shows that different parts of the AI (called modules) need different amounts of correction.

2. The "Module-Adaptive" Approach

MARR gives every single book (module) its own personalized volume knob.

Instead of a global "Volume 1.0," Book A might get "Volume 0.5," and Book B might get "Volume 1.2."
This allows the AI to find the perfect balance for each specific part: enough correction to fix the mistakes, but not so much that it creates new errors.

3. The "PID" Strategy (The Smart Thermostat)

How does the AI know what volume to set for each book without trying every single number (which would take forever)?

They use a strategy borrowed from engineering called PID Control (Proportional-Integral-Derivative). Think of this like a smart thermostat in your house:

The Goal: Keep the room at the perfect temperature (lowest error).
The Sensor: The thermostat checks the current temperature (the reconstruction error).
The Adjustment:
- If it's too cold, it turns the heat up.
- If it's too hot, it turns it down.
- PID is special because it doesn't just look at now; it looks at the trend (is it getting hotter fast?) and the history (has it been cold for a while?). This prevents the heater from blasting on and off wildly.

In MARR, the AI uses this "thermostat" logic to automatically tune the volume knob for each module. It makes tiny adjustments, checks if the error got better, and settles on the perfect setting very quickly.

The Results: Smaller Backpack, Same Brain

The authors tested this on famous AI models (like Llama) and image models (like ViT).

The Test: They tried to shrink the models to very small sizes (2-bit or 4-bit).
The Outcome: MARR kept the models much smarter than previous methods.
- For text models, it improved performance by up to 20%.
- For image models, it improved performance by up to 4.6%.
The Cost: It takes a little bit more time to "tune" the model before you use it (about 2 to 3 times longer than the previous best method), but once it's tuned, it runs just as fast and fits in the same small backpack.

Summary

The paper solves a problem where fixing AI mistakes with "correction notes" accidentally created new problems. Their solution, MARR, acts like a smart thermostat for every single part of the AI, automatically finding the perfect amount of correction for each piece. This allows us to shrink AI models down to tiny sizes without losing their intelligence.

Technical Summary: MARR (Module-Adaptive Residual Reconstruction)

Problem Statement
Low-bit post-training quantization (PTQ) is critical for deploying large language models (LLMs) and vision transformers (ViTs) under strict memory and bandwidth constraints. While reconstruction-based PTQ methods (e.g., GPTQ) effectively minimize local quantization errors, they often fail to account for errors accumulated from preceding layers in low-bit settings. Recent approaches, such as GPTAQ and ResComp, address this by introducing cross-layer residual terms to correct accumulated activation mismatches. However, the authors identify a critical limitation in these methods: the reliance on a Hessian-Approximation (HA) assumption.

Under the HA assumption, the reconstruction objective simplifies the task-loss Hessian to a diagonal matrix. The authors demonstrate that while residual terms effectively reduce accumulated errors, they simultaneously introduce additional Hessian-Approximation (HA) bias. This bias arises because the residual term interacts with the approximation gap between the true task-loss Hessian and the simplified local Mean Squared Error (MSE) metric. Consequently, a residual strength that is too strong can amplify this bias, leading to suboptimal quantization performance. Furthermore, the optimal balance between error correction and HA bias is module-dependent; a single global residual scaling coefficient is insufficient to handle the heterogeneity across different modules in a network.

Methodology
To address the trade-off between accumulated-error correction and residual-related HA bias, the paper proposes Module-Adaptive Residual Reconstruction (MARR). The core methodology consists of three components:

Scaled Residual Reconstruction:
The authors introduce a scaling coefficient, $\alpha$ , to modulate the residual term $r$ in the reconstruction objective. The objective function is formulated as minimizing $\|\Delta w \hat{X} - \alpha r\|_F^2$ . This allows the method to explicitly control the strength of the cross-layer information injected into each module. Theoretical analysis (Eq. 6) shows that the HA bias contains terms dependent on the residual $r$ ; thus, scaling $\alpha$ directly adjusts the magnitude of this bias relative to the error correction benefit.
Module-Specific Scaling:
Recognizing that the optimal $\alpha$ varies across modules due to differences in activation statistics and propagated residual magnitudes, MARR assigns a unique scaling coefficient $\alpha_m$ to each module $m$ . This replaces the global coefficient used in prior works (e.g., GPTAQ where $\alpha=1$ ).
PID-Based Adaptive Estimation:
To determine the optimal $\alpha_m$ without expensive exhaustive grid searches, the authors design a Proportional-Integral-Derivative (PID)-based adaptive update strategy.
- Feedback Signal: The module-level reconstruction error (MSE) serves as the feedback signal. Since the quantization process is non-smooth and discrete, gradient-based optimization is infeasible. Instead, the method uses a zero-order feedback approach.
- Update Mechanism: The algorithm treats $\alpha_m$ as an adjustable variable. It computes a deviation signal based on the finite-difference trend of the reconstruction error. This signal is processed through a PID controller (with proportional, integral, and derivative components) to generate an incremental update for $\alpha_m$ .
- Stability: The update includes a $\tanh$ -bounded deviation signal to prevent overshoot and an early stopping criterion based on the relative change of the objective function. In practice, the coefficient converges within a few iterations (typically $T=3$ ).

Key Contributions

Theoretical Insight: The paper identifies and analyzes the "two-sided effect" of residual reconstruction: while it reduces accumulated errors, it introduces HA-related bias. This explains why fixed, strong residuals can degrade performance.
MARR Framework: The proposal of a module-adaptive framework that assigns specific scaling coefficients to residuals, allowing each module to independently balance error correction and approximation bias.
Efficient Optimization: The design of a PID-based adaptive update strategy that efficiently estimates module-specific coefficients using reconstruction error feedback, avoiding the computational cost of exhaustive search.
Empirical Validation: Extensive experiments demonstrating that MARR consistently outperforms state-of-the-art residual reconstruction baselines (GPTAQ, ResComp) under low-bit quantization (W2A4, W3A3, etc.) across various LLMs (Llama2, Llama3) and ViTs (DeiT).

Experimental Results
The authors evaluated MARR on several models under weight-activation (W/A) and weight-only (W) quantization settings:

LLMs (W2A4): MARR achieved significant perplexity reductions on WikiText2 and C4 datasets. For instance, on Llama2-7b, MARR reduced perplexity by 14.94% compared to GPTAQ. On Llama3-8b, the reduction was 9.82%. In terms of downstream task accuracy, MARR showed consistent improvements, with gains up to 2.54% on average zero-shot accuracy for Llama2-7b.
ViTs: On ImageNet, MARR improved top-1 accuracy across DeiT-T, DeiT-S, and DeiT-B models. Notably, on DeiT-S under W3A3 quantization, MARR achieved a 4.61% relative gain over the GPTAQ baseline.
Weight-Only Quantization: Even when activations remain in high precision (e.g., W2A16), MARR demonstrated improvements, though the gains were sometimes more task-dependent, confirming that the residual control mechanism remains beneficial even when activation mismatch is reduced.
Efficiency: The PID-based update introduces a one-time PTQ overhead of approximately 2.1–2.7× relative to the baseline GPTAQ method. The authors note this is acceptable given the performance gains and the fact that it remains significantly faster than back-propagation-based reconstruction methods.

Significance and Claims
The paper claims that MARR provides a principled solution to the suboptimal performance of existing residual reconstruction methods in low-bit PTQ. By explicitly modeling the trade-off between accumulated-error correction and HA bias, and by adapting this balance at the module level, MARR achieves superior quantization accuracy without requiring retraining. The authors emphasize the generality of their approach, noting its effectiveness across both LLMs and ViTs, suggesting that controlling residual contribution is a fundamental requirement for pushing quantization limits below 4-bit. The work concludes that while the PID strategy adds computational overhead, it is a necessary and efficient mechanism to avoid the suboptimal performance caused by a "one-size-fits-all" residual strength.