🤖 machine learning

LegoLM: Structured Weight Sharing for Large Language Models

LegoLM is a structured weight-sharing framework that overcomes the distributional mismatch and outlier dominance failures of global weight sharing through data-free adaptations like scalar-block encoding, percentile-selective replacement, and boundary-layer protection, achieving near-lossless compression for large language models while significantly outperforming existing PTQ-8bit methods.

Original authors: Joseph Bingham

Published 2026-08-11

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Joseph Bingham

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to fit a massive, intricate library of knowledge into a tiny backpack. This is the daily struggle for "Large Language Models" (LLMs), the super-smart computer brains that write stories, solve math problems, and chat with us. These brains are built from billions of tiny numbers called "weights," which act like the synapses in a human brain, connecting ideas together. To make these models run on regular laptops or phones, scientists need to shrink them down without losing their smarts. One popular idea for shrinking them is "weight sharing." Think of it like a group of friends who all agree to wear the exact same shirt from a catalog instead of buying their own unique ones. Instead of storing a unique number for every single connection in the brain, you just store a small list of "standard shirts" (centroids) and a note saying, "Hey, this connection wears Shirt #5." It's a clever way to save space, but until now, it has been a disaster when applied to these giant AI brains.

The problem is that AI brains are messy. Unlike a simple image filter where every part looks the same, different layers of an AI brain have weights that are wildly different sizes. Trying to force them all to wear the same "standard shirts" is like trying to fit a giant elephant and a tiny mouse into the same pair of shoes; the elephant gets crushed, and the mouse floats away. This paper introduces a new method called LegoLM that fixes this broken idea. Instead of forcing a one-size-fits-all approach, LegoLM acts like a smart tailor. It realizes that most of the time, the standard shirts work fine, but every now and then, there are "outlier" weights—giant, weird numbers that are so different they would ruin the whole outfit if forced into a standard shirt. LegoLM's secret sauce is to leave those few weird weights exactly as they are, untouched, while compressing everything else. The result is a model that is squeezed down to a fraction of its original size but keeps almost all of its intelligence, working better than other methods that require expensive data to fix.

The Two Ways Weight Sharing Breaks

The author of this paper, Joseph Bingham, discovered that global weight sharing fails for two very specific, distinct reasons. They call these "failure modes," and understanding them is key to seeing why LegoLM works.

Failure Mode 1: The Scale Mismatch (The "Wrong Shoe Size" Problem)
Imagine you have a collection of weights from different layers of the AI. Some layers are "loud" (their numbers are large), and some are "quiet" (their numbers are small). If you try to group them all together and find a single average "shirt" for a whole block of them, you run into a math problem. The paper proves that if you try to compress a block of weights that are different sizes, the error grows linearly with the size of the block. It's like trying to average a marathon runner's stride with a toddler's step; no matter how many different shoes you add to your catalog, you can't make one shoe fit both perfectly if you force them to be the same size. The paper shows that this mismatch creates confusion so severe that the model's ability to predict text (measured as "perplexity") explodes into the millions, making the AI speak gibberish.

Failure Mode 2: Outlier Dominance (The "Giant" Problem)
This is the more dramatic failure. Even if you use single numbers (scalar blocks) instead of blocks, there are a few weights in the AI that are just huge compared to the rest. These are the "outliers." If you have a codebook with, say, 8 standard shirts, the biggest shirt in the catalog might be size XL. But what if there is a weight that is size 10XL? If you force that 10XL weight to wear the XL shirt, the distortion is catastrophic. The paper shows that as models get bigger (from 124 million parameters to 7.2 billion), these outliers become even more dangerous. On a smaller model, replacing these outliers might make the AI slightly confused. But on a giant model like Mistral-7B, replacing those few giant weights causes the model to collapse completely, with a quality drop of over 1,134,279%. It's as if removing a single keystone from a massive cathedral causes the whole thing to crumble into dust.

The LegoLM Solution: Smart Tailoring

LegoLM fixes these problems with three simple, data-free tricks. It doesn't need to look at any training data or retrain the model; it just rearranges the weights.

Scalar Encoding: Instead of grouping weights into blocks (which causes the scale mismatch), LegoLM treats every single weight as an individual. This eliminates the "wrong shoe size" problem because each weight is free to pick the closest standard shirt without being dragged down by its neighbors.
Percentile-Selective Replacement: This is the magic trick. The algorithm looks at all the weights and finds the ones that are furthest away from any standard shirt. These are the "outliers." Instead of forcing them to wear a shirt, LegoLM says, "You're too special; you stay exactly as you are." It preserves the top 1% of these weird weights in their original, high-precision form. The rest of the 99% get compressed into the small codebook.
Boundary-Layer Protection: The very first and very last layers of the AI are extra sensitive. LegoLM gives them a little extra care, though the paper notes this is less critical for the biggest models than the outlier protection.

The Results: Small Size, Big Brains

The author tested LegoLM on two models: GPT-2 Small (124 million parameters) and Mistral-7B (7.2 billion parameters). The results were surprising and impressive.

On Mistral-7B: Using a codebook of 128 standard values and preserving just 1% of the outlier weights, LegoLM compressed the model by 4.41 times. The result? The model's quality dropped by only 0.03%. This is essentially "lossless" compression. It actually performed better than a popular method called PTQ-8bit, which only compressed it 4 times and had slightly more error.
The Outlier Rescue: The most dramatic finding was about the outliers. When they tried to compress Mistral-7B by replacing all weights (even the giants) with standard values, the model's quality crashed by 1,134,279%. But by simply saving that tiny 1% of outliers, they rescued the model, bringing the error down to a manageable 14.24% even at a massive 9.74x compression.
Scale Matters: The paper found that the "outlier problem" gets much worse as models get bigger. On the small GPT-2, replacing outliers caused a 23% drop. On the giant Mistral-7B, it caused a 1,134,279% drop. This suggests that bigger models rely even more heavily on these few weird numbers to function.

Why This Matters

LegoLM is special because it is data-free. Most other compression methods need to feed the model thousands of example sentences to figure out how to shrink it. LegoLM just looks at the weights themselves and does the math. It can compress a 7-billion-parameter model in under 30 minutes on a single graphics card.

The paper concludes that the key to making weight sharing work isn't just having a better catalog of shirts; it's knowing when not to use the catalog. By identifying and protecting the few "giant" weights that hold the model together, LegoLM allows us to shrink these massive AI brains into backpacks without losing their minds. It turns a method that was previously broken for large models into a competitive, efficient tool for the future of AI.

Technical Summary: LegoLM – Structured Weight Sharing for Large Language Models

Problem Statement

Large language models (LLMs) require aggressive compression for practical deployment, as even 16-bit precision models exceed consumer GPU memory. While post-training compression methods like quantization, pruning, and low-rank factorization are established, weight sharing—where multiple weights share a single learned centroid value from a global codebook—has historically failed when applied to LLMs. Previous work successfully applied weight sharing to convolutional networks, but its application to transformers has lacked systematic study.

This paper identifies that global weight sharing fails for LLMs due to two distinct, theoretically grounded failure modes:

Distributional Mismatch: When weights are grouped into vector blocks ( $d \ge 2$ ) and clustered globally, the codebook must serve layers with heterogeneous weight scales (varying by up to 3 $\times$ ). A centroid trained on a pooled distribution imposes a scale-mismatch penalty that grows linearly with block dimension $d$ , resulting in perplexity in the millions regardless of codebook size.
Outlier Dominance: In scalar blocks ( $d=1$ ), while scale mismatch is eliminated, LLM weight distributions contain extreme-magnitude scalars (outliers). For small codebook sizes ( $K$ ), a fraction of weights ( $\approx 1/K$ ) lies beyond the outermost Lloyd-Max decision threshold. Replacing these outliers with the nearest centroid causes catastrophic quality loss that accumulates across layers. Crucially, this effect scales superlinearly with model size; full replacement degrades a 124M parameter model by ~23% but a 7.2B parameter model by over 1,000,000%.

Methodology: The LegoLM Framework

LegoLM is a structured, data-free compression framework designed to resolve both failure modes simultaneously without access to calibration data or gradients. It employs three key adaptations:

Scalar-Block Encoding ( $d=1$ ): To eliminate the $d$ -linear mismatch component, weights are treated as scalar blocks rather than vector blocks. This allows each weight to be independently assigned to its nearest centroid without committing to a fixed multi-dimensional scale, substantially reducing cross-layer reconstruction error.
Percentile-Selective Replacement: To address outlier dominance, the method identifies weights with the largest quantization error (distance to the nearest centroid) and preserves them verbatim (storing them as fp16 rather than replacing them with a centroid index). This acts as a zero-data proxy for magnitude-based outlier detection. The threshold is determined by a percentile $p$ (e.g., preserving the top 1% of errors).
Boundary-Layer Protection: The first and last transformer blocks are identified as disproportionately sensitive. For smaller models, these layers are skipped from compression to prevent error accumulation, though this mechanism is found to be less critical for larger models compared to selective replacement.

Compression Ratio Calculation:
The compression ratio (CR) accounts for the storage of the global codebook, the index stream (compressed via Huffman coding to reduce overhead), and the preserved outlier weights. The formula is:
$CR = \frac{(N_c + N_s) \cdot 32}{n_r \cdot \lceil \log_2 K \rceil + n_k \cdot 32 + K \cdot 32 + N_s \cdot 32}$
Where $N_c$ and $N_s$ are weights in eligible and skipped layers, $n_r$ are replaced weights, and $n_k$ are kept-exact outlier weights.

Key Contributions

Formal Failure Modes: The paper provides theoretical characterizations (Proposition 1 and Theorem 3) and empirical validation for two failure modes of LLM weight sharing: cross-layer distributional mismatch (vector blocks) and outlier dominance (scalar blocks).
The LegoLM Framework: A novel, data-free compression method combining scalar encoding, selective outlier preservation, and boundary protection. It includes exact formulas for compression ratios accounting for selective replacement.
Scale-Invariant Rescue Mechanism: The discovery that outlier dominance grows superlinearly with model scale. While full replacement catastrophically fails on 7B models, preserving just 1% of outlier weights rescues the model to near-lossless quality. This rescue mechanism is consistent across a 60 $\times$ parameter range (124M to 7.2B).
Mechanism Ablation: A controlled ablation study demonstrating that selective replacement is the dominant mechanism for data-free compression. Adding selective replacement to standard per-layer K-means achieves near-lossless quality (+0.05% PPL), matching the global LegoLM approach within 0.02 percentage points. The global codebook structure offers practical efficiency (storing 128 centroids vs. 4,096) but negligible quality differences when selective replacement is applied.

Experimental Results

The framework was evaluated on GPT-2 Small (124M), GPT-2 Medium (354M), and Mistral-7B (7.2B) using the WikiText-2 test set and downstream tasks (LAMBADA, HellaSwag).

Mistral-7B Performance:
- Near-Lossless: At $K=128, p=99\%$ , LegoLM achieves 4.41 $\times$ compression with only +0.03% perplexity (PPL) degradation. This outperforms PTQ-8bit (4.00 $\times$ , +0.04% PPL) in both quality and compression ratio.
- High Compression: At $K=8, p=99\%$ , it achieves 9.74 $\times$ compression with +14.24% PPL degradation, outperforming PTQ-4bit (8.00 $\times$ , +17.85% PPL).
- Downstream Accuracy: At $K=64, p=99\%$ (5.12 $\times$ compression), the model preserves downstream accuracy within measurement noise, matching PTQ-8bit accuracy while offering 1.28 $\times$ higher compression.
Comparison to Baselines:
- PTQ-8bit/4bit: LegoLM matches or exceeds their quality at comparable or higher compression ratios without requiring calibration data or Hessian computation.
- Deep Compression (Per-layer K-means): Without selective replacement, per-layer K-means fails catastrophically on Mistral-7B (e.g., +930,306% PPL at $K=256$ ). With selective replacement ( $p=99\%$ ), it recovers to near-lossless quality (+0.05%), confirming the centrality of outlier preservation.
- Pruning and Factorization: Unstructured magnitude pruning and SVD-based factorization fail catastrophically on Mistral-7B at high compression levels.
Scale Sensitivity:
- Full replacement ( $p=100\%$ ) at $K=128$ degrades GPT-2 Small by +23%, GPT-2 Medium by +79%, but Mistral-7B by +1,134,279%.
- Selective replacement ( $p=99\%$ ) maintains near-identical quality across all scales (+0.03% to +0.18% PPL degradation).
Phase Transition: A sharp transition in robustness occurs between $K=4$ (catastrophic failure) and $K=8$ (functional). At $K=8$ , preserving 1% of outliers is sufficient to rescue the model, whereas $K=4$ remains broken even with 5% preservation.

Significance and Claims

The paper claims that LegoLM establishes a new paradigm for data-free LLM compression by systematically addressing the theoretical reasons why global weight sharing previously failed. Its primary significance lies in:

Data-Free Efficiency: It achieves competitive or superior results to calibrated methods (like GPTQ) without requiring any calibration data, gradients, or Hessian computations, making it applicable in settings where data is proprietary or unavailable.
Theoretical Insight: It identifies outlier dominance as a scaling law where error accumulation grows superlinearly with model size, necessitating specific preservation strategies for large models.
Practical Viability: It demonstrates that a simple combination of scalar encoding and selective outlier preservation can compress 7B parameter models to ~4.5 $\times$ with negligible quality loss, and up to ~10 $\times$ with functional utility, all in under 30 minutes on a single GPU.

The author positions the "outlier dominance scaling law" and the "selective replacement" mechanism as independent contributions that may benefit other compression techniques (quantization, pruning) that currently treat all weights uniformly.