🤖 AI

LLM Probability Concentration: How Alignment Shrinks the Generative Horizon

This paper introduces the Branching Factor (BF) to demonstrate that alignment tuning significantly reduces LLM output diversity by concentrating probability distributions early in generation, a mechanism that enhances reasoning stability in Chain-of-Thought models and can be replicated in base models through strategic token prompting.

Original authors: Chenghao Yang, Sida Li, Ari Holtzman

Published 2026-03-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chenghao Yang, Sida Li, Ari Holtzman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) as a giant, magical choose-your-own-adventure book.

Every time the model writes a word, it's standing at a crossroads. It has to decide which path to take next. In the beginning, there are thousands of paths. Some lead to funny jokes, some to serious news, some to nonsense, and some to brilliant answers.

This paper, titled "LLM Probability Concentration: How Alignment Shrinks the Generative Horizon," investigates what happens when we "align" these models (train them to be helpful, harmless, and follow instructions). The authors discovered that alignment doesn't just make the model nicer; it fundamentally changes the map of the adventure, shrinking the number of paths available.

Here is the breakdown using simple analogies:

1. The "Branching Factor" (BF): How Many Doors Are Open?

The authors invented a metric called the Branching Factor (BF). Think of this as counting how many open doors the model sees at every step.

High BF (Base Model): The model is in a massive, chaotic hallway with 1,000 open doors. It can go left, right, up, down, or through a secret tunnel. It's very diverse, but also very unpredictable.
Low BF (Aligned Model): The model is in a narrow corridor with only 2 or 3 doors. Most of the time, one door is glowing brightly (the "correct" or "polite" answer), and the others are barely visible.

The Big Discovery: When you align a model (teach it to be a helpful assistant), you don't just paint the walls; you lock most of the doors. The model becomes much more predictable because it has fewer choices.

2. The "Super-Focused" Assistant

The paper found that alignment acts like a spotlight.

Before Alignment: The model is like a jazz musician improvising. It might play a beautiful note, or it might play a weird noise. It's creative but risky.
After Alignment: The model is like a strict tour guide. It knows exactly where the group is going. It doesn't wander off the path.

The authors found that aligned models are 2 to 5 times less diverse than their base versions. At the very start of a conversation, an aligned model might have 12 possible next words, but after alignment, it might only have 1.2. It's almost as if the model is saying, "I know exactly what to say next," before it even starts speaking.

3. Why "Temperature" Doesn't Work Anymore

In AI, we often use a setting called "Temperature" to make the model more creative (like turning up the chaos in a room).

The Old Way: If you turn up the temperature on a base model, it gets wild and creative because there are so many doors to choose from.
The New Reality: If you turn up the temperature on an aligned model, nothing much happens. Why? Because all the other doors are locked. Even if you tell the model to "be crazy," it can't find a path to get there. It's stuck in the narrow hallway.

4. The "Chain of Thought" Trick

You might have noticed that models using "Chain of Thought" (thinking step-by-step) seem very stable and reliable. The paper explains why:

Imagine the model is walking down a long hallway.
Early in the walk: There are still a few side paths.
Late in the walk: The hallway gets so narrow that there is literally only one way forward.
The Trick: Chain of Thought forces the model to walk a long distance before it gives the final answer. By the time it reaches the answer, it has already walked so far down the narrow hallway that it has no choice but to give the "correct" answer. It's not that the model is smarter; it's that it has run out of other options.

5. The "Nudge" Experiment: It Was There All Along

The most fascinating part of the paper is the conclusion about how this happens.

The Myth: Alignment rewrites the model's brain from scratch.
The Reality: The base model already knew how to be polite and focused; it just didn't choose to do it often.
The Proof: The researchers took a base model and just gave it a tiny "nudge"—a single word like "Sure" or "Here is the answer." Suddenly, the model's behavior changed. It immediately locked into that narrow, low-diversity path.

The Analogy: Imagine a dog that can sit, but usually runs around. You don't need to retrain the dog to sit; you just need to say "Sit." Once it sits, it stays there. Alignment is just teaching the model to say "Sit" to itself at the very beginning of every sentence.

Summary: The Trade-Off

The paper concludes that alignment is a trade-off:

Pros: The model is safer, more helpful, and gives consistent answers. It's great for customer service or math problems.
Cons: The model loses its "soul" and creativity. It becomes a "generative monoculture" where everyone sounds the same.

The Takeaway: If you want a model that is diverse and creative, you have to use the "base" model and accept the risk of it being weird. If you want a model that is reliable and safe, you use the "aligned" model, but you must accept that it has a very small "generative horizon"—it can only see a few steps ahead, and all of them look the same.

1. Problem Statement

Despite their impressive capabilities, aligned Large Language Models (LLMs) (those fine-tuned for helpfulness and safety via RLHF or instruction tuning) frequently exhibit a lack of output diversity and increased determinism compared to their base counterparts. While this consistency is often desirable for safety, it leads to:

Reduced sensitivity to decoding strategies (e.g., temperature, nucleus sampling).
"Homogenized" outputs that may stifle creativity.
A lack of rigorous metrics to quantify why and how this reduction in diversity occurs.

Existing metrics like token-level entropy or $n$ -gram diversity are either too local, confounded by vocabulary size, or fail to capture the global structure of the generation tree. The authors aim to rigorously conceptualize and measure this phenomenon, termed LLM Probability Concentration.

2. Methodology: The Branching Factor (BF)

To quantify probability concentration, the authors introduce the Branching Factor (BF), a token-invariant measure derived from information theory.

Definition: BF is defined as the length-normalized distribution perplexity (or exponentiated entropy rate).
$B \equiv \exp\left(\frac{1}{N} \tilde{H}(Y_{1:N}|x; \theta)\right)$
Where $\tilde{H}$ is the marginal entropy of the sequence. Conceptually, BF represents the effective number of plausible next-token choices available to the model at any given step, averaged over the generation.
Estimation: Calculating exact entropy is intractable for long sequences due to the exponential number of trajectories. The authors leverage a theoretical result (Theorem 3.1) showing that for typical sequences, the Negative Log-Likelihood (NLL) converges to the realized entropy as sequence length increases.
- They propose a hybrid estimator: using exact entropy for short sequences and NLL for long sequences to efficiently estimate BF without exhaustive tree search.
Experimental Setup:
- Models: Llama-2/3 families (Base vs. Instruct/Chat), OLMo-2, and Qwen models.
- Tasks: Reasoning (MMLU), Controlled Generation (Cognac), News Generation (BBCLatestNews), and Creative Story Generation.
- Variables: Prompt Complexity ( $C$ ), Model Size ( $S$ ), Model Generation ( $G$ ), and Alignment Tuning ($AT$).

3. Key Contributions & Findings

A. Alignment Drastically Reduces Branching Factor

Magnitude: Alignment tuning reduces BF by a factor of 2–5 overall. At the beginning of generation, the reduction can be an order of magnitude (e.g., dropping from 12 to 1.2).
Implication: Aligned models have significantly fewer viable branches to explore. This explains why they are less sensitive to decoding hyperparameters (like temperature); there are simply no alternative high-probability paths to select even when sampling is "loosened."

B. Dynamic Concentration During Generation

Trend: BF naturally decreases as generation progresses. Models "commit" to narrower trajectories as they generate more tokens.
Chain-of-Thought (CoT) Stability: CoT prompting works by extending the reasoning chain. This pushes the generation of the final answer into later, lower-BF stages of the tree. Because the model is already in a low-entropy, deterministic region, the final output becomes highly stable and less sensitive to sampling noise.

C. Mechanism: Steering to Latent Low-Entropy Paths

Hypothesis: Alignment does not fundamentally reshape the model's manifold but rather nudges the model toward specific stylistic tokens (e.g., "Sure," "Here is the answer") that unlock pre-existing low-entropy trajectories.
Evidence (Nudging Experiments): By forcing a base model to start with an aligned-style prefix (generated by a smaller aligned model), the authors observed an immediate and sharp drop in BF. This confirms that alignment acts as a "steering" mechanism toward latent low-entropy subspaces rather than creating new ones from scratch.

D. Pareto Analysis of Factors

Using Pareto analysis, the authors determined that Alignment Tuning is the primary driver of BF reduction, accounting for >80% of the variance in BF across tasks. This effect outweighs model size, generation version, and prompt complexity.

4. Key Results & Implications

Phenomenon	Explanation via BF
Decoding Insensitivity	Aligned models have low BF (few branches). Changing temperature or $p$ -value has little effect because the probability mass is already concentrated on 1–2 tokens.
CoT Stability	CoT forces the model to traverse deep into the tree where BF is naturally low, resulting in deterministic final answers.
Variance Reduction	Low BF correlates strongly with low variance in Majority Voting ($Maj@K$). Long-CoT models show the lowest variance.
Forking Risks	Resampling (forking) at late stages (low BF) leads to significant performance drops because the model is "locked in" to a specific trajectory; alternative paths are low-quality.

5. Significance & Future Directions

Diagnostic Tool: BF serves as a powerful, unified metric to diagnose LLM behavior, explaining disparate phenomena (diversity loss, decoding insensitivity, CoT stability) under a single probabilistic framework.
Training Insights: The findings suggest that current alignment methods (SFT, DPO) primarily function by narrowing the generative horizon. The paper notes that the specific stage (SFT vs. DPO) contributing most to BF reduction varies by model scale.
Societal Impact: The authors warn that this "probability concentration" may lead to societal homogeneity bias, where AI outputs reinforce a narrow set of views and stifle creativity.
Recommendations:
- Decoding: Simply tweaking decoding parameters is insufficient to restore diversity in aligned models; diversity loss is structural.
- Training: Future work should focus on training objectives that balance instruction-following with distributional diversity (e.g., diverse preference optimization) or system-level interventions (model collaboration).
- Application: Parallel sampling should be performed early in generation (when BF is high) to ensure meaningful diversity, rather than late-stage resampling.

In summary, the paper provides a rigorous mathematical explanation for the "stiffness" of aligned LLMs, demonstrating that alignment acts as a funnel, collapsing the vast generative space of base models into a narrow, highly predictable corridor.