💬 NLP

DecompressionLM: Deterministic, Diagnostic, and Zero-Shot Concept Graph Extraction from Language Models

The paper introduces DecompressionLM, a deterministic, zero-shot framework that leverages Van der Corput sequences and arithmetic decoding to extract concept graphs from language models without pre-defined queries, revealing that activation-aware quantization significantly expands concept coverage while uniform quantization causes severe collapse—a distinction not captured by standard perplexity metrics.

Original authors: Zhaochen Hong, Jiaxuan You

Published 2026-02-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhaochen Hong, Jiaxuan You

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Opening the Black Box

Imagine a Large Language Model (LLM) as a massive, locked library. Inside, it holds billions of facts and ideas it learned from the internet.

For a long time, researchers tried to see what was inside by asking the library specific questions, like "Who wrote Hamlet?" or "What is the capital of France?" This is like having a pre-written shopping list. If you only ask about things on your list, you'll never discover the rare, dusty books hidden in the back corners that you didn't know existed.

DecompressionLM is a new tool that stops using the shopping list. Instead, it throws the doors wide open and asks the library to simply "spit out everything it knows about a topic, one keyword at a time." The goal is to map out the entire library, not just the sections you already know about.

The Problem: The "Rich-Get-Richer" Trap

When you ask a standard AI to generate a list, it usually uses a method called Beam Search. Think of this like a group of hikers trying to find the best path up a mountain. They all stick together, following the path that looks the steepest and most promising.

The Issue: Because they all follow the same "best" path, they end up walking in a circle around the most popular, high-traffic trails (common concepts like "Torts" or "Contract Law"). They miss the quiet, overgrown trails where the rare, interesting concepts live (long-tail concepts).
The Result: The AI gives you a list that looks good but is actually very repetitive and misses the deep, diverse knowledge it actually possesses.

The Solution: The "Van der Corput" Compass

The authors created a new way to explore the library called DecompressionLM. Instead of sending a group of hikers who copy each other, they send out thousands of independent explorers, each with a unique, mathematically perfect compass.

No Shopping List: They don't tell the AI what to look for. They just say, "Give us legal concepts."
The Compass (Van der Corput Sequence): This is a special mathematical pattern that ensures the explorers spread out evenly across the entire map. They don't cluster together; they cover every inch of the territory, from the busy city center to the quiet wilderness.
Parallel Exploration: Because each explorer is independent, they don't need to talk to each other. You can run thousands of them at the same time on different computers without them getting confused.

The Discovery: What Happens When You Squeeze the Library?

The researchers tested this tool on models that had been "compressed" (shrunken down) to make them run faster on smaller computers. This is like taking a giant encyclopedia and trying to fit it into a pocket notebook.

They compared two ways of shrinking the book:

Method A (Uniform Squeeze): Imagine squishing the whole book evenly. The paper gets thin everywhere.
Method B (Smart Squeeze / AWQ): Imagine looking at the book and realizing some pages are just filler, while others have the most important diagrams. You squeeze the filler pages flat but leave the important diagrams thick and clear.

The Surprising Results:

The "Smart Squeeze" (AWQ): Even though the book was smaller, this method actually revealed more concepts than the original, uncompressed version! It was like the squeezing process somehow cleared the fog, allowing the AI to access rare, long-tail knowledge it was previously hiding.
The "Uniform Squeeze" (GPTQ): This method was a disaster. It didn't just shrink the book; it tore out the rare pages. The AI could still talk about common things, but it lost almost all its ability to generate rare concepts. The "map" of knowledge became fragmented and broken.

The Twist: The "Fluency" Lie

Usually, when we check if a compressed AI is working, we look at Perplexity (a score that measures how "confused" the AI sounds).

The Paper's Finding: The "Uniform Squeeze" models had great Perplexity scores. They sounded smooth and fluent when explaining a concept. But, when the researchers used DecompressionLM to see what concepts the model could actually come up with, they found the model had lost 70–80% of its knowledge.
The Analogy: It's like a person who can speak perfectly fluent English but has forgotten 80% of the vocabulary. They can talk smoothly about "the cat" and "the dog," but if you ask about "quantum physics" or "18th-century maritime law," they just can't find the words. The smooth talking hid the fact that the knowledge was gone.

The Hallucination Check

To make sure the AI wasn't just making things up, the researchers checked the generated legal concepts against a real database of US court cases.

The Result: The best models (top-ranked on standard tests) had very few "hallucinations" (fake concepts). The worst models had a huge gap: they were generating many concepts that didn't exist in real law.
The Takeaway: A model that scores well on standard tests but fails this "knowledge map" test is like a student who memorized the answers to a practice test but doesn't actually understand the subject.

Summary

DecompressionLM is a new way to "audit" AI models. It stops asking the AI specific questions and instead forces it to reveal its entire knowledge map. The paper shows that:

Standard ways of shrinking AI models (Quantization) can destroy their ability to access rare knowledge, even if they still sound fluent.
A specific "smart" shrinking method (AWQ) actually helps models access more diverse knowledge.
We need new ways to test AI that look at how much it knows, not just how well it talks about what it already knows.

Technical Summary: DecompressionLM

Problem Statement

Existing methods for probing knowledge in Large Language Models (LLMs) rely on pre-defined queries (e.g., cloze-style prompts or specific templates). This "closed-world" approach limits extraction to concepts researchers already know to ask about, failing to reveal the full scope of a model's encoded knowledge. Furthermore, standard decoding-based probing methods suffer from three critical inefficiencies:

Cross-sequence coupling: Sequential exploration (e.g., beam search) concentrates probability mass on high-frequency prefixes, suppressing diverse continuations.
Competitive decoding effects: Autoregressive sampling creates a "rich-get-richer" dynamic where early high-probability choices dominate, reducing coverage of long-tail concepts.
Scalability constraints: Exhaustive probing via sequential context expansion incurs quadratic time complexity due to self-attention mechanisms, making comprehensive knowledge extraction computationally prohibitive.

Additionally, standard evaluation metrics like perplexity often fail to capture the breadth of knowledge or the fidelity of factual grounding, particularly in compressed (quantized) models.

Methodology: DecompressionLM

The authors introduce DecompressionLM, a stateless, zero-shot framework designed to systematically sample an LLM's output distribution to discover concept graphs without pre-specified queries.

Core Mechanisms

Van der Corput (VdC) Sampling with Arithmetic Decoding:
- Instead of sequential beam search, the method employs deterministic quasi-random sampling using Van der Corput low-discrepancy sequences.
- Combined with arithmetic decoding, this maps sequences to real numbers in $[0, 1)$ . The VdC sequence generates an unbounded, low-discrepancy set of codes ( $v_n$ ) that ensure uniform coverage of the output space.
- This approach enables fully parallel, stateless generation. Each sequence is generated independently from the same prefix prompt using a unique code, eliminating cross-sequence coupling and allowing computation to be offloaded to different machines without coordination.
Concept Extraction Pipeline:
- Prompting: Models are prompted to generate domain-relevant concepts as a simple list of keywords (one per line), avoiding explanatory text to minimize structural assumptions.
- Normalization: Raw concepts undergo Unicode normalization, lowercasing, ASCII filtering, and morphological singularization (e.g., "violations" $\to$ "violation").
- Merging: A fuzzy matching algorithm (Levenshtein distance with length-based blocking) merges near-duplicates to construct a canonical concept set.
- Graph Construction: A directed graph is built where nodes are concepts and edges represent weak co-occurrence signals derived from consecutive lines in the generated sequences.
Validation:
- For legal domains, extracted concepts are verified against CourtListener, a database of U.S. case law, to distinguish factual concepts from hallucinations.

Key Contributions

Systematic Coverage via VdC Sampling: The paper replaces sequential beam search with deterministic quasi-random sampling. This provides reproducible, order-independent, seed-free coverage schedules, enabling consistent benchmarking across models and quantization variants.
Stateless Concept Extraction: By decoupling sequences, the method avoids biases introduced by dynamic context accumulation and enables scalable, embarrassingly parallel generation.
Concept Coverage as a Diagnostic Metric: The authors propose "concept coverage" (breadth, connectivity, and truthfulness of extractable knowledge) as a complementary evaluation dimension to traditional perplexity and downstream task benchmarks.

Experimental Results

The framework was evaluated across two model families (Qwen2.5-7B and Llama-3.1-8B), five quantization variants (BF16, GPTQ-Int8, GPTQ-Int4, AWQ-4bit, BNB-4bit), and four domains (US Law, Git, Radiology, FAA Aviation).

Quantization Effects

AWQ-4bit (Activation-aware): Surprisingly, this method expanded concept coverage by 30–170% compared to the BF16 baseline while maintaining high graph connectivity. This suggests that protecting salient weights based on activation statistics preserves access to long-tail knowledge.
GPTQ-Int4 (Uniform): This method induced a catastrophic collapse, reducing concept coverage by 71–86% and causing severe graph fragmentation (e.g., only 40.8% of concepts in the largest connected component for Llama).
Perplexity Decoupling: Standard perplexity metrics remained relatively stable across quantization variants (except for the degenerate GPTQ-Int4 regime), failing to reflect the massive divergence in concept coverage. This indicates that a model can generate fluent text for known concepts while losing the ability to surface rare, valid concepts.

Robustness and Hallucination

Offset Stability: Experiments with multiple VdC offsets confirmed that the observed degradation patterns (e.g., AWQ superiority vs. GPTQ collapse) are intrinsic to the model's knowledge structure and not artifacts of initialization.
Hallucination Gap: In the US Law domain, a 19.6-point gap in hallucination rates was observed between top-ranked and bottom-ranked models on the MMLU-Pro Law benchmark. Top models averaged ~15% hallucination, while bottom models averaged ~35%.
Frequency Correlation: Some models exhibited "head corruption," where even frequently generated concepts became unreliable (negative frequency-hallucination correlation), while others showed "tail collapse," where rare concepts were hallucinated but common ones remained grounded.

Significance and Claims

The paper claims that DecompressionLM establishes a new diagnostic capability for assessing LLMs, particularly in the context of deployment-ready compressed models.

Beyond Perplexity: The findings challenge the primacy of perplexity as a quality metric for compressed models. A model may maintain low perplexity while losing 70% of its extractable concepts, a failure mode invisible to standard metrics but critical for retrieval-augmented generation, knowledge-intensive QA, and creative applications.
Quantization Insights: The work reveals that different quantization methods affect distinct aspects of model knowledge. Activation-aware quantization (AWQ) appears to preferentially preserve the ability to generate diverse, long-tail concepts, whereas uniform quantization degrades this capability despite similar perplexity scores.
Evaluation Dimension: Concept coverage is proposed as a necessary, complementary dimension for evaluating the factual grounding and knowledge breadth of models intended for real-world deployment.

The authors conclude that this framework allows for the extraction of comprehensive concept graphs without prior knowledge of the domain, lowering the barrier to auditing model knowledge and revealing qualitative properties of knowledge representation that traditional benchmarks miss.