🤖 machine learning

On the Quantization Robustness of Diffusion Language Models in Coding Benchmarks

This paper demonstrates that diffusion-based language models exhibit superior quantization robustness compared to their autoregressive counterparts at low bitwidths, suggesting they offer significant advantages for efficient deployment in coding tasks.

Original authors: Aarav Gupta, Gururaj Deshpande, Chandreyi Chakraborty

Published 2026-04-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Aarav Gupta, Gururaj Deshpande, Chandreyi Chakraborty

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have two brilliant chefs trying to cook the same complex meal (writing computer code).

Chef A (The Autoregressive Model, like Qwen3) is a traditional chef. They cook one ingredient at a time, in a strict order. They are incredibly fast and precise, but if you ask them to cook a huge feast, they need a massive kitchen (memory) and it takes a long time because they can't multitask.
Chef B (The Diffusion Model, like CoDA) is a modern, experimental chef. Instead of cooking step-by-step, they start with a bowl of random noise and gradually "clean" it up, refining the whole dish simultaneously until it's perfect. They are great at seeing the big picture, but they also need a big kitchen.

The Problem: Both chefs are too expensive to run on a small home stove (like a laptop or a phone). They use too much memory and electricity.

The Solution: Quantization. Think of this as "downgrading the quality of the ingredients" to save space. Instead of using premium, high-precision spices (32-bit or 16-bit numbers), we try to use cheaper, lower-quality spices (4-bit or 2-bit numbers). The goal is to shrink the recipe so it fits on a small stove without ruining the taste of the meal.

The Big Experiment

The researchers wanted to see: Which chef handles "cheap ingredients" better?

They took both Chef A (Qwen3) and Chef B (CoDA) and forced them to cook using these cheap, low-quality spices. They tested them on two famous cooking challenges: HumanEval and MBPP (which are basically tests to see if the code they write actually works).

The Surprising Results

1. The "Collapse" Effect

Chef A (Qwen3) was very sensitive. When they switched to 4-bit spices, the food got a bit salty. But when they went down to 3-bit or 2-bit, the dish completely fell apart. It was like trying to bake a cake with sand instead of flour; the structure just couldn't hold.
Chef B (CoDA) was surprisingly tough. Even with the cheap 4-bit spices, the dish still tasted great. It only started to fall apart when they went to the very cheapest 2-bit spices.
The Takeaway: The Diffusion Chef (CoDA) is much more "resilient." It can survive on a budget diet much better than the traditional Chef.

2. The Speed Race

At the start (High Quality): Chef A was slightly faster. Their step-by-step method is very optimized for high-end kitchens.
On a budget (Low Quality): Here is the twist. When the ingredients got cheap (low precision), Chef B actually became faster than Chef A.
- Why? Chef A still had to do all the complicated setup work (like measuring every single spice individually) even if the spices were cheap. Chef B, however, just did one big "cleaning" pass. Once the ingredients were cheap, the overhead of Chef A's complex setup became the bottleneck, while Chef B's simple method shined.

3. The "Smart Mix" (HAWQ)
The researchers also tried a fancy technique called HAWQ. Imagine you have a budget, but you want to keep the most important parts of the dish premium while making the less important parts cheap.

They analyzed which parts of the model were "sensitive" (like the delicate sauce) and which were "robust" (like the sturdy pot).
They gave the "sauce" layers high-quality spices (16-bit) and the "pot" layers cheap spices (4-bit).
Result: This created a perfect balance. You could shrink the model to fit on a phone without losing much taste, finding the "sweet spot" between size and quality.

The Bottom Line

The paper suggests that Diffusion Language Models (like CoDA) might be the future for running AI on small devices.

While traditional models (like Qwen3) are currently the kings of performance, they are fragile when you try to shrink them down. Diffusion models are like "tank-like" structures; they might be a bit slower to start, but they are incredibly robust. If you need to run a coding assistant on a laptop with limited memory, a Diffusion model might handle the "budget mode" much better than a traditional one.

In short: If you want the best performance on a supercomputer, use the traditional chef. But if you want to run a smart coding assistant on a small, energy-efficient device, the Diffusion chef is the one who won't break a sweat when you switch to the cheap ingredients.

1. Problem Statement

Large Language Models (LLMs), particularly autoregressive (AR) models like GPT-4 and Qwen, excel at coding tasks but suffer from high memory footprints and inference latency due to sequential token generation. While Diffusion-based Language Models (d-LLMs) offer an alternative with bounded inference costs via iterative denoising, their behavior under Post-Training Quantization (PTQ) remains largely unexplored.

The core research question is: Are diffusion-based LLMs more robust to aggressive low-bit quantization (2–4 bits) compared to their autoregressive counterparts? Existing PTQ techniques (like GPTQ and HAWQ) are well-established for AR models but have not been systematically evaluated against d-LLMs, leaving a gap in understanding which generation paradigm is better suited for resource-constrained, quantized deployment.

2. Methodology

The authors conducted a controlled empirical comparison between a diffusion-based coding model (CoDA 1.7B) and its autoregressive counterpart (Qwen3-1.7B). Both models share the same architecture (Transformer) and parameter count (1.7B) but differ in training objectives (diffusion vs. autoregressive) and training data.

Experimental Setup

Models: CoDA 1.7B (Diffusion) vs. Qwen3-1.7B (Autoregressive).
Benchmarks: HumanEval and MBPP (Mostly Basic Programming Problems), evaluating pass@1 and pass@5.
Hardware: NVIDIA L40S and H100 GPUs.
Quantization Methods:
1. GPTQ (Group-wise Quantization): A weight-only PTQ method using an inverse Hessian approximation to minimize activation reconstruction error. Applied to bitwidths of 2, 3, 4, and 8 bits.
2. HAWQ (Hessian-Aware Quantization): A mixed-precision approach assigning bitwidths based on layer sensitivity (derived from Hessian eigenvalues).
  - Adaptation: The authors modified the original HAWQ algorithm (designed for smaller CNNs) to handle 1.7B parameter models. They implemented finite differences to approximate Hessian-vector products (due to Flash Attention incompatibility) and sparse sampling ( $\rho=0.1$ ) to make sensitivity calculations computationally feasible.
  - Strategy: Layers were sorted by sensitivity, and a predefined split ratio (e.g., 50% 16-bit, 50% 8-bit) was applied greedily.

Evaluation Metrics

Accuracy: Pass rates on coding benchmarks.
Latency: Mean time per denoising step (CoDA) vs. per token generation (Qwen3).
Memory: Model size reduction via quantization.

3. Key Contributions

Standardized Comparison: Established the first empirical framework comparing PTQ robustness between a diffusion LLM and its direct autoregressive counterpart under identical evaluation pipelines.
Robustness Discovery: Demonstrated that diffusion models exhibit superior resilience to low-bit quantization (2–4 bits) compared to AR models, which suffer catastrophic performance drops at these bitwidths.
Mixed-Precision Pareto Frontier: Showed that HAWQ-derived mixed-precision configurations for diffusion models provide smoother trade-offs between accuracy, latency, and memory, enabling fine-grained deployment optimization.

4. Key Results

A. Accuracy Robustness (GPTQ)

Low-Bit Performance: CoDA maintained functional accuracy down to 4-bit precision. In contrast, Qwen3 experienced a "catastrophic collapse" at 3-bit and near-total failure at 2-bit.
Degradation Magnitude:
- CoDA (4-bit): Accuracy drop of only 1.8 to 5 points across benchmarks.
- Qwen3 (4-bit): Accuracy drop of 12 to 23.2 points.
- Observation: At 4-bit, CoDA retained significantly more of its original capability than Qwen3.
Hypothesis: The authors suggest that Qwen3's higher data-to-parameter ratio (trained on 36T tokens vs. CoDA's 180B) may have led to "over-training," making it more sensitive to quantization noise, whereas CoDA's training dynamics (diffusion objective) inherently provided better robustness.

B. Latency and Efficiency

Baseline: The 16-bit autoregressive Qwen3 was slightly faster (26.8ms) than CoDA (28.3ms) due to highly optimized causal masking kernels.
Quantized Performance:
- At 4-bit and 8-bit, CoDA became competitive with or faster than Qwen3.
- Reasoning: As quantization reduces arithmetic costs, fixed system-level overheads (e.g., kernel launch, token handling) dominate latency. CoDA's single-step denoising pass over the full sequence handles these overheads more efficiently than the sequential overhead of AR token generation when arithmetic is cheap.
- CoDA achieved 25–40% faster generation latency than Qwen3 at 4-bit and 8-bit precisions.

C. Mixed-Precision (HAWQ)

Two-Level Splits: Configurations splitting layers into two precision groups (e.g., 50% 16-bit / 50% 8-bit) successfully tracked the Pareto frontier, allowing users to target specific memory constraints with minimal accuracy loss.
Three-Level Splits: Attempts to mix three precisions (16/8/4) resulted in model collapse, likely due to the inability of the model to self-correct imprecise weights without fine-tuning.

5. Significance and Conclusion

The paper provides compelling evidence that Diffusion Language Models may be inherently more robust to quantization than autoregressive models. This has profound implications for efficient deployment:

Resource-Constrained Edge Deployment: Diffusion models can be deployed on devices with limited VRAM (using 4-bit or even 3-bit quantization) while maintaining coding performance, whereas AR models of the same size fail at these bitwidths.
Latency Efficiency: Contrary to the belief that diffusion is slower, the study shows that under quantization, diffusion models can outperform AR models in latency due to reduced sensitivity to system overheads.
Future Directions: The authors recommend future work involving training both paradigms on identical datasets to isolate architectural effects, exploring calibration datasets specific to coding (e.g., OpenCoder), and investigating Quantization-Aware Training (QAT) to recover performance in complex mixed-precision scenarios.

In summary, this work challenges the dominance of autoregressive models for low-resource scenarios, suggesting that diffusion-based architectures offer a promising, quantization-resilient alternative for efficient coding assistants.