💬 NLP

Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMs

This paper identifies that diversity collapse in large language models is primarily caused by order and shape miscalibration in their probability distributions during decoding, rather than by limitations in sampling heuristics, leading to a trade-off between output validity and diversity.

Original authors: Amin Banayeeanzade, Qingchuan Yang, Dhruv Tarsadiya, Fatemeh Bahrani, Leonardo Blas, Alfy Samuel, Robin Jia, Meisam Razaviyayn, Sai Praneeth Karimireddy

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Amin Banayeeanzade, Qingchuan Yang, Dhruv Tarsadiya, Fatemeh Bahrani, Leonardo Blas, Alfy Samuel, Robin Jia, Meisam Razaviyayn, Sai Praneeth Karimireddy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you ask a Large Language Model (LLM) to tell a story, write a poem, or suggest a random city. You expect a wide variety of creative answers. Instead, the model often gets stuck in a rut, giving you the same few clichés over and over again (like "Once upon a time..." or "Valparaíso, Chile").

This paper investigates why this happens. The authors argue that the problem isn't just that the model is "boring" or that our current tools for picking words (sampling methods) are bad. Instead, the problem is built into how the model organizes its own internal probability map. They call this a "Calibration Bottleneck."

To explain this, imagine the model is a Librarian standing in front of a massive wall of books (all possible next words). The Librarian has to pick the next word to continue a story.

The Two Main Problems

The paper identifies two specific ways this Librarian is "miscalibrated" (misaligned with reality), which causes the diversity to collapse.

1. Order Calibration: The Librarian Can't Sort the Books

The Analogy:
Imagine you ask the Librarian to pick a "good" book. You expect the Librarian to put all the "good" books at the very top of the shelf and all the "bad" books at the bottom.

What the model does: The Librarian puts some good books at the top, but then mixes in a bunch of bad books right in the middle. Then, further down the shelf, there are more good books hidden among the bad ones.
The Consequence: If you tell the Librarian, "Just give me the top 10 books," you might get 5 good ones and 5 bad ones. If you say, "Give me the top 100 to be safe," you get 90 bad books.
The Paper's Claim: The model fails at Order Calibration. It cannot reliably rank "valid" (correct/meaningful) words higher than "invalid" (nonsense) words. Because valid and invalid words are mixed together in the ranking, any tool that tries to filter the list (like "Top-K" or "Top-P") is forced to make a terrible choice: either cut off good ideas to avoid bad ones, or include too many bad ideas to save the good ones.

2. Shape Calibration: The Librarian is Obsessed with One Book

The Analogy:
Now imagine the Librarian does manage to separate the good books from the bad ones. But when looking at the "Good" pile, the Librarian is obsessed with just one specific book.

What the model does: The Librarian puts 90% of their attention on one specific "good" book (e.g., "Once upon a time"). The other 99 "good" books get only a tiny, tiny sliver of attention. Meanwhile, the "bad" books are scattered in a long, thin tail at the very bottom.
The Consequence: Even if you try to shake things up (by using "Temperature," which is like telling the Librarian to be more random), the Librarian just shifts their obsession slightly. They might stop obsessing over "Once upon a time" and start obsessing over "In a world where...", but they still ignore the other 98 good options. Worse, making the Librarian more random often just makes them pick a "bad" book from the long tail instead of a rare "good" one.
The Paper's Claim: The model fails at Shape Calibration. The probability is too concentrated on a few specific valid options, leaving the rest of the valid options starved of attention.

The Domino Effect: Why One Mistake Becomes a Disaster

The paper explains that these problems get worse the longer the story gets.

The Analogy:
Imagine you are walking through a maze.

Step 1: At the first turn, you have a 90% chance of picking the right path and a 10% chance of picking a dead end. That seems fine.
Step 2: At the next turn, you again have a 90% chance of being right.
The Result: If you have to make 20 turns, the chance of getting the entire path right without hitting a single dead end drops to almost zero.

The paper proves mathematically that because the model makes small errors at every single step (picking the wrong mix of words or focusing too much on one word), these errors compound. By the time the model finishes a sentence or a paragraph, the chance of it having explored a truly diverse, valid path is exponentially tiny.

What Did They Do?

The researchers didn't just guess; they built a "diagnostic toolkit":

Controlled Tests: They gave the models simple tasks where they knew the exact list of correct answers (like generating random numbers or naming US states).
The "Oracle" Test: They created a perfect filter that knew exactly which words were valid. Even with this perfect filter, the models still struggled to be diverse because of the "Shape" problem (the obsession with one word).
The Verdict: They tested 14 different models (from small to huge). They found that bigger models didn't fix the problem. Even the smartest models still had these sorting and concentration issues.

The Bottom Line

The paper concludes that the lack of diversity in AI isn't just a bug in the "picking" tools (like temperature or top-k sampling). It is a fundamental flaw in how the model distributes its confidence.

It can't sort valid words above invalid ones reliably.
It can't spread its attention evenly across all the valid words.

Until the model learns to organize its internal library differently, simply tweaking the settings won't unlock true creativity or diversity. The "bottleneck" is the model's own distribution, not the tools we use to read it.

Technical Summary: Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMs

Problem Statement

Despite the critical role of diversity in applications ranging from creative generation to scientific discovery, modern Large Language Models (LLMs) frequently exhibit "mode collapse," generating outputs that collapse into a narrow subset of plausible responses. While prior work has established benchmarks for measuring this lack of diversity, the underlying causes within the inference-time probability distributions remain poorly understood. Standard sampling heuristics (e.g., temperature scaling, top-k, top-p, min-p) often fail to recover meaningful diversity; they either shift probability mass toward invalid or nonsensical continuations or truncate valid alternatives. This paper posits that the bottleneck is not merely a limitation of decoding strategies but a fundamental property of the LLM's conditional distribution itself, specifically regarding how probability mass is allocated across valid and invalid tokens.

Methodology

The authors introduce a validity–diversity framework to analyze the trade-off between generating valid outputs and exploring diverse valid alternatives. They decompose the diversity collapse into two complementary forms of miscalibration: Order Calibration and Shape Calibration.

1. Theoretical Framework

Validity: Defined as the total probability mass assigned to the set of valid responses ( $V$ ).
Diversity: Defined as the normalized effective support size (exponential of entropy) of the model distribution restricted to the valid set.
Order Calibration: The assumption that valid tokens are reliably ranked above invalid tokens. The paper argues this is frequently violated, with valid tokens interleaved throughout the ranked distribution.
Shape Calibration: The assumption that probability mass is distributed uniformly across valid tokens. The paper argues distributions are often overly concentrated on a few valid tokens (sharp head) with a heavy tail of mixed valid/invalid tokens.

2. Empirical Diagnostics

To probe these bottlenecks, the authors developed controlled diagnostic tools:

Token Validity Labeling: Using an LLM-as-a-judge (Qwen3.5-35B) to evaluate the validity of greedy completions initiated by specific candidate tokens. This allows for the construction of precision-recall curves for cutoff strategies.
Controlled Testbeds: Utilizing tasks with exactly known valid sets (e.g., random number generation, naming US states) to compute exact validity and diversity metrics without relying on noisy semantic judgments.
Oracle Baselines: Implementing "oracle" cutoff strategies that know the exact number of valid tokens at each step but not their identity, isolating the impact of rank-based filtering from the underlying distribution shape.
Multi-step Analysis: Extending single-step diagnostics to depth $d > 1$ to observe how local errors compound over sequence generation.

Key Contributions

Validity–Diversity Framework: A formal analysis of the trade-off at both token and sequence levels, attributing diversity collapse to distributional properties rather than just decoding heuristics.
Identification of Calibration Bottlenecks:
- Order Miscalibration: Valid tokens are not consistently ranked above invalid ones. This creates an inherent trade-off for any rank-based filtering rule: expanding the cutoff to recover valid tokens inevitably admits invalid ones, while tightening it excludes valid alternatives.
- Shape Miscalibration: Probability mass is non-uniformly concentrated on a small number of valid continuations, while the tail contains a heavy mix of valid and invalid tokens. Temperature scaling flattens the head but disproportionately increases the mass of the invalid tail.
Theoretical Proofs of Compounding Loss:
- Order: Theorem 4.2 demonstrates that local precision-recall trade-offs compound multiplicatively. Even small, constant local failures in separating valid/invalid tokens lead to an exponential decay in the reachable valid set as sequence length increases.
- Shape: Theorem 5.2 shows that under geometric decay assumptions, maintaining high validity requires a sharp distribution, which inherently reduces entropy (diversity) over valid tokens. This diversity loss compounds exponentially with the number of branching steps ( $m$ ).
Empirical Diagnostics: Development of controlled settings (random number generation, state naming) and oracle baselines to systematically trace the source of diversity failures across 14 language models of varying families and scales.

Results

Order Calibration Failure: Experiments across 14 models (Qwen, Llama, Olmo families) show that valid tokens are frequently interspersed with invalid ones in the ranked distribution. The precision-recall trade-off worsens significantly as decoding depth increases. Even the optimal cutoff strategy cannot perfectly separate valid from invalid tokens.
Shape Calibration Failure: In controlled tasks (e.g., random number generation), increasing temperature improves diversity only at the cost of validity. The invalid tail grows faster than the recovery of rare valid alternatives.
Scaling Does Not Solve the Problem: While larger models show a mild improvement in order calibration (higher AUC of precision-recall frontiers), they do not eliminate the trade-off. Recall at high precision remains low and non-monotonic across model sizes.
Decoding Heuristics are Insufficient: No standard top-token filtering method (top-k, top-p, min-p) effectively recovers diversity. Even an oracle cutoff (knowing the exact count of valid tokens) fails to achieve ideal diversity in tasks with shape miscalibration, indicating that the distribution's shape, not just the rank, is the limiting factor.
Oracle Intervention: Applying an oracle filter (sampling only from tokens labeled valid by the diagnostic) significantly improves both embedding diversity and lexical diversity (lower Self-BLEU), proving that valid alternatives exist in the distribution but are missed by standard rank-based rules.

Significance and Claims

The paper claims that diversity collapse is a consequence of order and shape miscalibration in the LLM distribution, rather than a failure of specific sampling algorithms.

Hardness Result: The authors establish a theoretical hardness result for any decoding strategy relying solely on top-token filtering: if valid and invalid tokens are interleaved, maintaining high sequence-level validity necessitates sacrificing a constant fraction of valid continuations at each step, leading to exponential collapse in diversity.
Diagnostic Value: The proposed framework provides a lens to diagnose why existing methods fail, moving beyond black-box benchmarking to understanding the mechanistic sources of diversity loss.
Implications for Future Work: The findings suggest that future solutions must move beyond tuning decoding hyperparameters. Effective mitigation may require:
- Developing calibration-aware decoders that incorporate auxiliary validity signals beyond probability rank.
- Investigating training objectives (pretraining, alignment, safety) that affect distribution sharpness and rank ordering to preserve broader valid support without sacrificing quality.

The authors maintain a modest stance, noting that their work diagnoses the bottleneck rather than solving it, and that their controlled experiments serve as diagnostics for specific distributional mechanisms rather than a complete account of all diversity failures in complex, open-ended generation.