💬 NLP

Lost in Sampling: Assessing Lexical Reachability in LLMs via the Word Coverage Score (WCS)

This paper introduces the Word Coverage Score (WCS), a metric demonstrating that standard decoding sampling filters in Large Language Models systematically prune contextually appropriate, low-frequency human vocabulary, thereby acting as unintended censorship mechanisms that suppress lexical diversity and homogenize generated text.

Original authors: Samer Awad, Javier Conde, Carlos Arriaga, Tairan Fu, Javier Coronado-Blázquez, Pedro Reviriego

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Samer Awad, Javier Conde, Carlos Arriaga, Tairan Fu, Javier Coronado-Blázquez, Pedro Reviriego

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) as a master storyteller sitting in a vast, infinite library. This library contains every word in the human language, from common ones like "the" and "and" to rare, sophisticated, and beautiful words like "ephemeral" or "petrichor."

According to this paper, even though the storyteller has access to the entire library, the way they choose their next word is like having a very strict, over-protective editor standing over their shoulder. This editor doesn't just help; they actively cut off paths, forcing the storyteller to only use the most common, safe, and boring words.

Here is a breakdown of what the paper found, using simple analogies:

1. The "Garden of Forking Paths"

The authors compare the model's decision-making process to a famous story by Jorge Luis Borges about a garden where every possible future path exists simultaneously.

The Ideal: In this garden, the storyteller could choose any path, including the winding, rare ones that lead to unique and colorful expressions.
The Reality: The model's "sampling filters" (the settings used to pick words) act like a chainsaw. Before the storyteller can even look at the interesting, rare paths, the chainsaw cuts them all down. The model is forced to walk only on the wide, paved main roads where the most common words live.

2. The "Word Coverage Score" (WCS): A Survival Test

To prove this, the researchers invented a test called the Word Coverage Score (WCS).

The Setup: They took a list of "Middle-Long Tail" words—words that aren't super common (like "the") but aren't nonsense (like "flibber"). These are the words that give human writing its flavor and texture.
The Test: They placed these words into real sentences written by humans. Then, they asked the AI: "If you were writing this sentence, would your current settings allow you to pick this specific word?"
The Result: The answer was often no. Even though the AI knew the word (it was in its training data), the mathematical rules it uses to pick words (like Top-p or Top-k) mathematically blocked the word from ever being chosen. The word was effectively "erased" from the AI's vocabulary for that moment.

3. The "Alignment" Trap

The paper found that making AI "safer" and more helpful (a process called alignment or instruction tuning) actually made the problem worse.

The Analogy: Imagine a wild, creative artist (the "Base" model) who uses a huge variety of colors. Then, a manager (the "Alignment" process) tells the artist, "Please stick to the colors the customers like best."
The Outcome: The artist starts using even fewer colors. The "Instruct" or "Aligned" versions of the models tested in the paper erased more words than the raw, un-tuned versions. They became even more homogenized, speaking in a flatter, more repetitive tone.

4. The "Default Settings" Problem

The researchers checked the settings that companies recommend for everyday use (like chatting or writing emails).

The Finding: Under these standard settings, a huge chunk of the "Middle-Long Tail" words (between 22% and 57% in some models) were completely unreachable.
The Metaphor: It's like giving a chef a kitchen full of exotic spices but telling them, "You can only use salt, pepper, and ketchup." The chef might know what saffron or truffle oil is, but the rules of the kitchen prevent them from ever using it.

5. The Long-Term Danger: A "Flattened" World

The paper warns about a dangerous cycle.

The Cycle: If AI models start producing text that only uses common words, and then future AI models are trained on that AI-generated text, the "rare" words will disappear entirely from the training data.
The Result: The "Garden of Forking Paths" will eventually become a single, straight, boring hallway. The rich, diverse texture of human language could be permanently smoothed out into a homogenized, repetitive sludge.

Summary

The paper argues that the tools we use to make AI text "coherent" and "safe" are accidentally acting as censors. They are mathematically pruning the unique, rare, and expressive parts of human language, leaving us with a version of English that is safe, consistent, but incredibly boring. The authors suggest we need to rethink how we pick words from the model to keep the "garden" alive and full of diverse paths.

Technical Summary: LOST IN SAMPLING

Problem Statement

Modern Large Language Models (LLMs) frequently generate text that is homogeneous, repetitive, and lacking the lexical richness of human discourse. While previous research has attributed this "mode collapse" to model alignment techniques (such as RLHF or DPO) or training data limitations, this paper investigates the role of decoding mechanics in suppressing linguistic diversity. The authors posit that standard sampling filters (e.g., Top-p, Top-k, Min-p) act as aggressive pruning mechanisms that mathematically eliminate low-frequency, high-information words from the generative path, even when those words reside within the model's latent probability space. This results in a "structural collapse" where the model's operational vocabulary is artificially constrained, smoothing unique human expression into a homogenized discourse.

Methodology: The Word Coverage Score (WCS)

To quantify the extent to which sampling filters prune contextually appropriate vocabulary, the authors introduce the Word Coverage Score (WCS). This metric measures the "lexical survival rate" of human words as a function of sampling parameters. The evaluation follows a four-stage Forced-Path Audit:

Lexical Selection: A target set of "Middle-Long Tail" words is selected from the Google Web Trillion Word Corpus. These words are ranked between 10,000 and 40,000 in frequency to avoid common functional words and extremely rare noise.
Contextual Pairing: Target words are mapped back to naturalistic contexts extracted from the PG-19 dataset (books published before 1919), ensuring evaluation within a linguistically rich, human-authored environment free from modern web-crawled homogenization.
Forced-Path Audit: For each word $w$ $w$ (composed of sub-word tokens $t_1, \dots, t_n$ $t_{1}, \dots, t_{n}$ ) and context $C$ $C$ , the model is provided with the ground-truth prefix. Instead of sampling, the system records the rank and probability of the ground-truth tokens. The audit determines if these tokens would survive the pruning logic of specific sampling algorithms:
- Top-k Survival: The token must rank within the top $k$ most probable.
- Top-p (Nucleus) Survival: The token must belong to the smallest set of tokens whose cumulative probability meets threshold $p$ .
- Min-p Survival: The token's probability must exceed a scaled fraction $m$ of the maximum token probability.
Metric Calculation: The WCS is calculated as the aggregate mean reachability ( $R$ ) across the target word set $W$ and their contexts. A word is considered "covered" ( $R=1$ ) only if every constituent token survives the filter at its respective step.

Experimental Configuration

The study evaluated open-weight models with fewer than 20 billion parameters, comparing Base (raw) variants against Instruct/It (aligned) variants. The model families included Llama, Mistral, Qwen, Gemma, and DeepSeek. The experiments systematically swept sampling parameters:

Top-p: 0.7 to 0.99
Top-k: 1 to 20
Min-p: 0.01 to 0.1
Temperature: 0.7, 1.0, and 1.5

The evaluation focused on the fraction of words that become unreachable ( $WCS(\theta, w) = 0$ ) across 10 contexts per word, effectively measuring "word erasure."

Key Results

Systemic Word Erasure: Under standard industry-default settings (e.g., Top-p $\approx$ 0.8–0.95, $T \approx$ 0.7), a significant fraction of target words (22% to 57%) are completely unreachable in any of their contexts. For example, the Gemma-4-E4B-it model erased 57% of target words under its default configuration.
Impact of Alignment: Instruction-tuned (aligned) models generally exhibit lower lexical reachability than their base counterparts. Across matched pairs, mean word-level reachability dropped from 0.740 (Base) to 0.728 (Instruct), and the aggregate WCS decreased from 0.290 to 0.277. This suggests that preference optimization processes directly restrict the usable vocabulary space.
Distillation Deficit: The DeepSeek-R1-Distill-Qwen-14B model showed significant vocabulary loss compared to its base Qwen2.5-14B, indicating that distilling structured reasoning capabilities introduces additional constraints on lexical accessibility.
Ineffectiveness of High Temperature: Even under aggressive high-temperature settings ( $T=1.5$ ), a substantial fraction of contexts failed to select the target words, indicating that simply increasing temperature does not fully recover lexical diversity and often compromises coherence.
Model-Specific Variance: While most families showed reduced diversity after alignment, the Gemma family displayed non-uniform patterns; Gemma-3-12B-it improved over its base, whereas Gemma-4-E4B-it suffered severe erosion, highlighting that alignment impacts vary by architecture and training iteration.

Significance and Claims

The paper claims that standard sampling defaults act as unintended censorship mechanisms, mathematically pruning the "Garden of Forking Paths" (a metaphor for the model's latent probability space) before a token is ever selected. The WCS provides a rigorous diagnostic tool to quantify this phenomenon, revealing that the homogenization of LLM output is not solely a result of training data or alignment but is structurally enforced by decoding filters.

The authors argue that this lexical erosion poses a long-term risk to language evolution. As model-generated text populates public repositories and becomes training data for future models, the recursive ingestion of this flattened discourse could permanently constrict the active human lexicon, erasing the "Long Tail" of diverse expression. The paper concludes that preserving vocabulary diversity requires a paradigm shift: treating lexical breadth as a core architectural objective equivalent to factual correctness, rather than a superficial parameter, and developing semantic-guided decoders that can distinguish between noise and contextually rich, low-frequency vocabulary.

Limitations

The authors acknowledge several constraints:

Scope: The study is limited to open-weight models <20B parameters and English language only.
Data Bias: Contexts are drawn from pre-1919 literature (PG-19), introducing a historical bias, though this avoids modern web homogenization.
Binary Metric: The WCS measures absolute exclusion ( $R=0$ or $1$) and does not capture "soft erosion" where a word remains reachable but its probability is significantly dampened.
Tokenizer Variance: Differences in sub-word tokenization across model families mean that the number of filtering steps for a single word varies, affecting direct comparability.
Static Context: The audit uses a fixed 256-token prefix, not assessing long-range dependencies.