🤖 machine learning

ReRound: Reconstructive Rounding to Resolve Midpoint Ambiguity in Calibration-Free LLM Quantization

ReRound is a calibration-free post-training quantization method that leverages a conditional diffusion model to resolve midpoint rounding ambiguities in low-bit LLM weights, consistently outperforming standard techniques for 3-bit and 4-bit quantization while maintaining offline efficiency.

Original authors: He-Yen Hsieh, H. T. Kung

Published 2026-08-12

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: He-Yen Hsieh, H. T. Kung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to shrink a giant, incredibly detailed library of knowledge down to fit inside a tiny, portable backpack. This is the world of Large Language Models (LLMs), the super-smart AI brains that can write stories, solve math problems, and chat like humans. These brains are made of billions of tiny numbers called "weights," and to make them fast and cheap to run on regular phones or laptops, scientists try to squeeze these numbers into smaller boxes. This process is called quantization.

Think of the original, full-size numbers as high-resolution photos. To save space, we want to turn them into low-resolution pixel art. The standard way to do this is called "Round-to-Nearest" (RTN). It's like looking at a number and asking, "Is this closer to 1 or 2?" If it's 1.4, you round down to 1. If it's 1.6, you round up to 2. It's a simple, automatic rule. But there's a tricky spot: what if the number is exactly 1.5? Or very close to it, like 1.48? At this "midpoint," the decision is ambiguous. Rounding down or up creates almost the same amount of error, so the simple rule just picks one arbitrarily. In a massive library of billions of numbers, making thousands of these tiny, arbitrary guesses can accidentally scramble the meaning of the whole book, making the AI dumber.

This is where a new method called ReRound comes in. Instead of blindly guessing at the tricky midpoints, ReRound acts like a detective that looks at the surrounding neighborhood of numbers to figure out the best choice. It uses a special kind of AI training called a "diffusion model" (the same tech used to generate images from text) to learn the hidden patterns of how numbers are arranged in the AI's brain. When it sees a number stuck at a midpoint, it asks, "Based on the company this number keeps, should it really be rounded down, or does it belong up?" By using these learned patterns to guide the rounding, ReRound can fix the mistakes of the standard method without needing to retrain the whole AI or feed it new examples. It's a clever way to squeeze more accuracy out of a smaller backpack, making powerful AI accessible to everyone without needing expensive supercomputers.

The Detective's Guide to Better AI

In the world of artificial intelligence, the quest to make models smaller and faster often hits a wall: the "midpoint problem." When scientists try to shrink the massive numbers inside an AI (a process called quantization), they usually use a simple rule: if a number is closer to the floor, round down; if it's closer to the ceiling, round up. But what happens when a number is right in the middle of the room? Standard rules just flip a coin. The paper ReRound: Reconstructive Rounding to Resolve Midpoint Ambiguity in Calibration-Free LLM Quantization argues that this coin flip is a missed opportunity.

The authors, He-Yen Hsieh and H. T. Kung from Harvard University, propose a new strategy called ReRound. Instead of blindly rounding numbers near the middle, ReRound uses a "diffusion prior"—a learned guess about what the original, perfect numbers looked like before they were shrunk. Imagine you have a blurry, pixelated photo of a cat. A standard rounding rule might just guess the color of a fuzzy pixel based on its immediate neighbors. ReRound, however, uses a trained AI (specifically a U-Net diffusion model) to "hallucinate" or reconstruct what the entire patch of the photo should look like, based on the patterns it learned from the original, high-quality image.

Here is how the magic happens, step by step:

The Training Phase: Before the AI is even shrunk, ReRound takes a peek at the full-size, high-precision weights of the model. It chops these weights into tiny 64x64 patches and trains a diffusion model to predict the original, full-size patch just by looking at a low-bit, blurry version of it. Think of this as teaching a detective to recognize the texture of a specific fabric just by feeling a tiny, fuzzy swatch.
The Reconstruction: When it's time to shrink the model, ReRound doesn't just round the numbers. It runs the blurry, low-bit numbers through its trained detective (the diffusion model) to generate a "reconstructed" version of the weights. This isn't the final weight; it's a guide. It's like the detective saying, "I know this fuzzy pixel is near a midpoint, but based on the pattern of the whole patch, it should be a bit higher."
The Smart Rounding: ReRound only uses this detective's advice when the standard rule is truly confused (near the midpoint). If the number is clearly close to the floor or ceiling, it sticks to the standard rule. But if the number is wobbly near the middle, ReRound checks the reconstructed value. If the reconstruction suggests rounding the other way, and the "distance" from the midpoint isn't too large, ReRound flips the switch.
The Safety Check: To make sure it doesn't flip too many switches and break the model, ReRound generates a few different versions of the shrunken model, each with a slightly different "tolerance" for how much it trusts the detective. It then picks the winner by looking at the "skeleton" of the matrix (its singular values). It chooses the version that keeps the most important structural patterns of the original, full-size model intact.

The results are impressive, especially for smaller AI models. The paper shows that ReRound consistently beats the standard "Round-to-Nearest" method when compressing models to 3-bit and 4-bit precision. For example, on models like Gemma 2 2B and Gemma 3 1B, ReRound improved accuracy by 0.1 to 1.3 points across various language tasks. In some cases, it even outperformed methods that required "calibration data" (feeding the AI thousands of example sentences to learn how to round), all while ReRound used no extra data at all. It worked entirely offline, using only the model's own weights.

The authors are careful to note that this isn't a magic bullet for every single problem. The method works best when the quantization parameters (like the scale and zero-point) are already fixed. It also requires training a separate diffusion model for each specific AI, which takes some time and computing power upfront (about 2 to 3 hours of training and a few hours of inference per model). However, once that is done, the actual process of shrinking the model is fast, taking only seconds to minutes.

Crucially, ReRound doesn't change how the AI runs on your phone or laptop. It only changes the final integer numbers stored in the memory. This means the AI runs just as fast as before, but with a bit more brainpower preserved. The paper suggests that this approach of using "reconstructed weights" as a guide for rounding could be a new standard for making AI smaller and smarter, proving that sometimes, the best way to make a decision isn't to look at the number in isolation, but to ask the neighborhood what it thinks.

Technical Summary: ReRound – Reconstructive Rounding to Resolve Midpoint Ambiguity in Calibration-Free LLM Quantization

1. Problem Statement

Post-training quantization (PTQ) reduces the storage and memory footprint of Large Language Models (LLMs) by mapping full-precision weights to low-bit integers. A critical step in this process is the final rounding assignment, where a quantized floating-point weight must be assigned to either its lower (floor) or upper (ceiling) adjacent integer.

Standard Round-to-Nearest (RTN) schemes make this decision based solely on scalar distance. While this is unambiguous for weights near integer boundaries, it creates midpoint ambiguity for weights near the center of quantization intervals (e.g., $r \approx 0.5$ ). In these regions, the scalar error incurred by rounding up or down is nearly identical. Consequently, RTN decisions in these ambiguous zones are often arbitrary, yet a small set of such decisions can significantly alter the overall rounding pattern of a linear layer, potentially degrading model accuracy. Existing methods either optimize quantization parameters (scales, zero-points) or use calibration data to learn rounding assignments, but few address the specific ambiguity of the rounding step itself without relying on activation samples or downstream labels.

2. Methodology: ReRound

ReRound is a calibration-free framework that revisits midpoint-ambiguous RTN decisions using information learned directly from the pretrained LLM's own weights. The method consists of three primary stages:

A. Conditional Weight Reconstruction via Diffusion

ReRound trains a conditional diffusion model (specifically a U-Net) on the pretrained LLM's weight matrices.

Training: The model learns to reconstruct full-precision weight patches ( $P$ ) conditioned on their low-bit stochastic rounding (SR) versions ( $P_{low}$ ). The target is the continuous full-precision patch, while the condition is the quantized observation.
Inference: During the PTQ phase, the model generates continuous reconstructed weights ( $W_{rec}$ ) conditioned on the deterministic RTN quantized weights ( $P_{cond}$ ).
Purpose: These reconstructed weights are not deployed as model parameters. Instead, they serve as a learned prior reflecting the distribution of weights within the specific LLM, providing evidence to resolve rounding ambiguities.

B. Reconstruction-Guided Rounding with Position-Dependent Tolerance

ReRound uses the reconstructed weights to propose alternative rounding assignments (switching from floor to ceiling or vice versa) only when the original RTN decision is ambiguous.

Proposal Condition: A change is proposed only if the reconstructed weight ( $\tilde{w}_{rec}$ ) rounds to the opposite adjacent integer ( $q_{alt}$ ) compared to the original RTN assignment ( $q_{rtn}$ ).
Position-Dependent Tolerance: Not all proposals are accepted. ReRound employs a tolerance metric $\tau_{pos}(r; \tau)$ $τ_{p os} (r; τ)$ based on the fractional position $r$ $r$ of the weight within its quantization interval.
- Near the midpoint ( $r \approx 0.5$ ), the tolerance is high, allowing reconstruction-guided changes.
- Near the integer boundaries, the tolerance decreases exponentially, preserving the standard RTN decision where the scalar error difference is significant.
- A change is accepted only if the deviation between the reconstructed and original weight falls within this position-dependent tolerance.
Candidate Generation: By sweeping a tolerance parameter $\tau$ , ReRound generates a set of candidate quantized integer weight matrices ( $Q_\tau$ ), each representing a different set of accepted rounding revisions.

C. Spectral Selection

To select the optimal candidate without calibration data, ReRound uses a weight-only spectral criterion.

Each candidate matrix $Q_\tau$ is de-quantized to form a continuous matrix $W_\tau$ .
The leading singular values of $W_\tau$ are compared to those of the original full-precision matrix $W$ .
The candidate with the smallest spectral discrepancy (closest match in dominant singular values) is selected as the final quantized weight matrix. This ensures the selected rounding pattern preserves the dominant structural modes of the original layer.

3. Key Contributions

ReRound Framework: A novel calibration-free approach that utilizes a conditional diffusion model trained on the model's own weights to guide midpoint-ambiguous rounding decisions.
Position-Dependent Tolerance: A metric that dynamically controls the acceptance of reconstruction-guided changes, prioritizing the midpoint region while preserving RTN near boundaries.
Spectral Selection Criterion: A method for selecting the optimal quantized integer matrix based on singular value preservation, eliminating the need for activation samples, text calibration, or downstream labels.
Compatibility: The method operates entirely offline, modifies only the quantized integer assignments (leaving scales, zero-points, and inference procedures unchanged), and can refine RTN or parameters from other PTQ methods.

4. Experimental Results

The paper evaluates ReRound on a range of small LLMs (e.g., Gemma 2 2B, Gemma 3 1B, Qwen3 1.7B, OLMo 2 1B) for 3-bit and 4-bit weight quantization.

vs. Calibration-Free Baselines: ReRound consistently outperforms standard group-wise RTN and other calibration-free methods (HQQ, BNB FP4, Hadamard-transformed RTN) across all tested models. It achieves the highest average accuracy on 3-bit quantization for all models and matches or exceeds the best results on 4-bit quantization for most.
vs. Calibration-Dependent Baselines: ReRound surpasses calibration-dependent methods like GPTQ, AdaRound, and SignRound on several models (e.g., achieving 67.4% average accuracy on Gemma 2 2B vs. 67.0% for SignRound) without using any calibration data.
Ablation Studies: Removing any component (reconstruction guidance, position-dependent tolerance, or spectral selection) results in lower accuracy, confirming the necessity of the full pipeline.
Generalizability: ReRound improves RTN baselines even when applied to quantization parameters generated by other methods (e.g., SINQ), demonstrating its utility as a post-hoc refinement tool.
Efficiency: The method introduces no overhead during low-bit inference. The offline cost involves one diffusion training run per model and a fast PTQ selection step (seconds per model).

5. Significance and Claims

The paper positions ReRound as a new paradigm for low-bit quantization that leverages structural patterns in pretrained weights as an effective prior. Its primary significance lies in:

Resolving Ambiguity: It specifically targets the "midpoint ambiguity" problem, a known limitation of standard RTN, using learned priors rather than heuristic rules.
Calibration-Free Efficiency: It achieves state-of-the-art accuracy among calibration-free methods and competes with calibration-dependent approaches, removing the need for costly calibration datasets.
Preservation of Inference: By only altering the integer assignments while keeping the quantization grid and inference logic identical to RTN, it ensures seamless deployment.

The authors note limitations, including the requirement for a separate diffusion model per LLM (introducing model-specific offline costs) and the current focus on small LLMs. They suggest future work could explore transferring weight priors across models and optimizing quantization parameters jointly with rounding.