Technical Summary: Where Reasoning Breaks Under Compression

Problem Statement

Current practices for fitting Large Language Models (LLMs) onto commodity hardware rely heavily on uniform quantization, where every layer of a model is compressed to the same bit-width (e.g., 4-bit or 3-bit). This approach is a matter of convenience rather than principle. A critical open question remains: Does reasoning capability depend on specific layers more than others? If certain layers are more critical for chain-of-thought arithmetic, uniform compression may waste precision on robust layers while starving fragile ones. Furthermore, it is unknown whether the "fragility" of layers is consistent across different model architectures or if it varies by family.

Methodology: The Quantization Autopsy

The authors introduce a diagnostic method called a "quantization autopsy" to probe layer-specific fragility without altering the model architecture or removing weights.

1. Localized Intervention: Starting from a full-precision "teacher" model, the authors construct a "student" model where a single contiguous block of layers is quantized to a low bit-width (2-bit or 3-bit), while all other layers remain at full precision (16-bit).
2. Sweeping: This damaged block is swept across the network in non-overlapping segments (typically dividing the model into six blocks).
3. Breakage Metric: The models are evaluated on GSM8K (grade-school math word problems). For every problem the full-precision teacher solves correctly, the system records whether the damaged student fails to solve it. This "break rate" serves as the metric for fragility.
4. Hardware & Scale: Experiments were conducted on a single Apple M3 Ultra workstation (256 GB unified memory) without a GPU cluster, testing three foundation models from independent lineages:
   • Qwen3-30B-A3B (30.5B params, 48 layers, MoE)
   • gpt-oss-20B (20.9B params, 24 layers, dense/banded)
   • Llama-4-Scout-109B (109B params, 48 layers, MoE)
5. Quantizer: A pure-software, blockwise, uniform symmetric quantizer was used. Notably, the 2-bit setting yields three effective levels (-1, 0, +1), making it a harsh perturbation designed to reveal structural weaknesses.

Key Results

1. Architecture-Dependent Fragility Geometries

The study reveals that compression fragility is real and layer-localized, but the "shape" of fragility differs significantly across model families. No single map of fragile layers transfers between models:

• Qwen3-30B (U-Shape): Fragility is concentrated at the network boundaries. The input-adjacent block (layers 0–8) and output-adjacent block (layers 40–48) show high break rates (77% and 85%, respectively), while the interior is robust (23–42% break rate).
• gpt-oss-20B (Inverted-U): Fragility is concentrated in the interior. The input is robust (5% break), but the mid-to-late interior peaks at 64% break rate, recovering slightly toward the output.
• Llama-4-Scout-109B (Front-Robust/Back-Fragile): The first block is entirely robust (0% break), while all subsequent layers (middle and late) are saturated (100% break).

2. Severity Scales with Model Size

The level of damage required to reveal these geometries tracks with model scale and redundancy:

• Smaller models (gpt-oss-20B): Saturate (fail everywhere) at 2-bit; structure only emerges at 3-bit.
• Medium models (Qwen3-30B): Structure emerges at 2-bit; 3-bit is too mild to show differences (flat).
• Larger models (Scout-109B): Flat at 3-bit; structure only emerges at the harsh 2-bit level.
  This suggests that larger, more redundant models require more aggressive compression to expose layer-specific dependencies.

3. Budget-Dependent Utility of Mixed Precision

The practical value of the fragility map is strictly budget-dependent:

• At 4-bit Budget: Uniform quantization already preserves accuracy (85.0% on GSM8K). Map-guided mixed precision offers no advantage and can even collapse if the robust interior is compressed too aggressively (e.g., to 2-bit) while the fragile ends are protected, due to error compounding.
• Below 4-bit Budget: Uniform quantization collapses (3-bit uniform drops to 41.7%). However, a map-guided recipe that protects the fragile boundary layers (keeping them at 4 or 5 bits) while compressing the robust interior to 3-bit retains the 4-bit accuracy (85.0%) at an average of 3.33 bits.
• Implication: Localization is redundant when precision is plentiful but decisive when it is scarce. The map allows a model to fit on a device with ~17% less memory (12.7 GB vs. 15.25 GB for the 30B model) while maintaining the accuracy of a larger uniform 4-bit model.

Significance and Claims

The paper claims three primary contributions:

1. A Diagnostic Method: A transparent, model-agnostic "quantization autopsy" that identifies where reasoning breaks under controlled damage, runnable on a single workstation for models up to 109B parameters.
2. Architecture-Specific Fragility: Evidence that compression fragility is not universal; it manifests as distinct geometries (U-shape, Inverted-U, Front/Back split) depending on the model family, scale, and training lineage.
3. A Budget-Dependent Prescription: The finding that mixed-precision allocation guided by fragility maps is only beneficial below the "uniform cliff" (typically below 4-bit). In this regime, protecting fragile boundary layers allows for smaller model footprints without accuracy loss, a regime critical for deployment on memory-constrained devices.

The authors emphasize that their results are descriptive of fragility under controlled damage and do not claim to be a production-ready quantization algorithm. They note limitations, including the modest size of the evaluation pools (8–26 problems per condition) and the use of a software simulation rather than hardware kernels, which limits claims about inference latency. The study concludes that while localization offers no benefit at generous bit budgets, it becomes a crucial tool for pushing models into the aggressive compression regimes required for edge deployment.