💬 NLP

Where does output diversity collapse in post-training?

This paper demonstrates that output diversity collapse in post-trained language models is primarily determined by training data composition rather than generation formats or specific post-training methods, revealing that the loss of semantic variety is embedded in model weights during supervised fine-tuning and cannot be resolved at inference time alone.

Original authors: Constantinos Karouzos, Xingwei Tan, Nikolaos Aletras

Published 2026-04-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Constantinos Karouzos, Xingwei Tan, Nikolaos Aletras

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, chaotic artist (the Base Model) who can paint in a million different styles, tell a million different stories, and solve problems in a million different ways. They are wild, creative, and full of surprises.

But then, you hire a manager to train them for a specific job. You want them to be helpful, safe, and follow instructions perfectly. So, you put them through a rigorous training camp (Post-Training).

The paper asks a simple but crucial question: Does this training make the artist better, or does it turn them into a robot that only paints the exact same picture over and over again?

The answer, according to this research, is: It turns them into a robot, and the reason depends entirely on what you fed them during training, not just how you trained them.

Here is the breakdown of their findings using some everyday analogies:

1. The Three Training Camps (The Lineages)

The researchers didn't just look at one model; they watched three different groups of artists go through the same three-step training process (Supervised Fine-Tuning, then Preference Optimization, then Reinforcement Learning).

The "Think" Group: These artists were trained by two specific teachers who were very good at explaining their work step-by-step (Chain-of-Thought). They were told, "Here is exactly how to solve this math problem, write down every step, and then give the answer."
The "Instruct" Group: These artists were trained by a huge crowd of different teachers (thousands of sources) who gave broad instructions but didn't always show the step-by-step work.
The "RL-Zero" Group: These artists skipped the classroom entirely. They were just thrown into a video game where they got points for winning and lost points for losing, with no human teachers showing them how to play.

2. Where Did the Creativity Die? (The Collapse)

The researchers found that the "creativity" (diversity) of the artists didn't die all at once. It died at different times depending on the group:

For the "Think" Group: Their creativity died immediately in the first class (SFT). Because they were only taught by two teachers who thought alike, the artists stopped trying new things right away. They learned to mimic those two teachers perfectly. By the time they got to the later training stages, there was no creativity left to kill.
For the "Instruct" Group: They kept their creativity for the first class. But then, in the second class (DPO), their creativity got crushed. This stage was designed to teach them what humans prefer. Since the training data was broad, the model tried to find the "average" human preference and ended up smoothing out all the unique edges, leaving a bland, uniform output.

The Lesson: It's not the method of training that kills creativity; it's the ingredients (the data). If you feed a model a narrow diet (two teachers), it gets narrow ideas early. If you feed it a broad diet, it gets narrow ideas later when you try to "optimize" it for human taste.

3. The "Thinking" Trick (Chain-of-Thought)

There was a popular theory that models were becoming boring because they were forced to "think out loud" (write a long reasoning chain before answering). People thought, "If we just make them answer directly without thinking, they'll be creative again!"

The researchers tested this by forcing the "Think" models to stop writing their reasoning and just give the answer.

The Result: Nothing changed. The answers were still boring and identical.
The Analogy: Imagine a chef who learned to cook only one specific recipe because their two teachers only taught them that one dish. If you tell the chef, "Stop writing down your shopping list and just give me the food," they still only know how to make that one dish. The problem wasn't the shopping list (the format); the problem was that they only learned one recipe (the weights).

The Takeaway: The lack of creativity is baked into the model's brain during training. You can't fix it at the end by changing how the model speaks.

4. The "Good" vs. "Bad" Boring

The researchers also looked at why the outputs became boring. They split the "boringness" into two types:

Type A: Getting Rid of Mistakes. Imagine a student taking a test. If they write 16 different answers, and 15 of them are wrong, and only 1 is right, the group of answers looks "less diverse" because the wrong ones are gone. This is actually good. It means the model is smarter.
Type B: Getting Rid of Good Ideas. Imagine a student who knows 16 different correct ways to solve a problem, but they only write down the exact same one every time. This is bad. It means the model has lost its ability to be creative even when it's right.

The study found that for math and coding, the "boringness" was mostly Type A (getting rid of wrong answers), which is fine. But for creative writing and opinions, the "boringness" was Type B (getting rid of different correct perspectives), which is dangerous because it makes the model sound like a single, homogenized voice.

Summary: What Should We Do?

If you are building AI or using it:

Don't just blame the training algorithm. If your model is boring, look at your training data. Did you use too few teachers? Did you use a narrow dataset?
Don't try to "fix" it by changing the output format. Making the model skip its reasoning steps won't bring back the creativity. The damage is already done.
Mix your ingredients. To keep AI creative, you need to feed it data from many different sources and styles, not just a few "perfect" examples.

In short: You can't teach a pigeon to fly like an eagle just by changing the cage; you have to give it the right food and environment to begin with. The "boringness" of modern AI isn't a glitch; it's a feature of how we've been feeding it.

1. Problem Statement

Post-trained Large Language Models (LLMs) exhibit a phenomenon known as output diversity collapse, where they generate significantly more uniform outputs compared to their base counterparts. This homogenization undermines inference-time scaling methods (e.g., self-consistency, pass@k sampling) and risks erasing valuable variation in creative and value-laden tasks.

Prior research has largely attributed this collapse to specific post-training algorithms (e.g., Direct Preference Optimization - DPO, or Supervised Fine-Tuning - SFT) without disentangling the effects of:

Training data composition (narrow vs. broad sources).
Generation formats (e.g., Chain-of-Thought reasoning vs. direct answers).
Model weights (learned distributions).

The authors aim to answer two critical questions:

Does diversity collapse depend on the post-training method or the data composition?
Is the collapse caused by the Chain-of-Thought (CoT) generation format at inference, or is it embedded in the model weights?

2. Methodology

The study utilizes the Olmo 3 model family (7B scale) and tracks output diversity across three parallel post-training lineages starting from the same base model:

Think Lineage: Trained on narrow, synthetic CoT data from two teachers (QwQ-32B and DeepSeek-R1). Follows the SFT $\to$ DPO $\to$ RL pipeline.
Instruct Lineage: Trained on broad, multi-source data (GPT-3.5/4/4.1) with reasoning traces stripped. Follows the same SFT $\to$ DPO $\to$ RL pipeline.
RL-Zero Lineage: Applies Reinforcement Learning (RL) directly to the Base model, bypassing SFT and DPO entirely.

Experimental Design:

Tasks: 15 diverse tasks including summarization, code generation, mathematical reasoning, instruction following, creative writing, and value pluralism.
Metrics: Four complementary diversity metrics:
- EAD: Lexical diversity (Expectation-Adjusted Distinct n-grams).
- SBERT: Semantic diversity (Mean pairwise cosine distance of sentence embeddings).
- NLI: Logical diversity (Entailment/Contradiction scores).
- Vendi Score: Effective number of distinct modes (based on eigenvalue entropy).
Key Interventions:
- CoT Suppression: For the "Think" lineage, the authors forced direct answers by suppressing CoT reasoning at inference (prefilling empty tags) to test if the format itself constrains diversity.
- Quality-Filtered Decomposition: Diversity was decomposed into a quality-control component (removal of incorrect answers) and a residual component (genuine narrowing among correct answers).

3. Key Contributions

Data Composition Drives Collapse Location: The study demonstrates that the stage at which diversity collapses depends on the training data, not just the algorithm.
- Think (Narrow Data): Collapses primarily during SFT (losing ~62% of base diversity) due to the narrow two-teacher distillation. DPO has minimal further effect.
- Instruct (Broad Data): SFT preserves more diversity; the major collapse occurs during DPO (~23% additional loss), which aggressively narrows the distribution.
Collapse is in Weights, Not Format: Suppressing CoT at inference for "Think" models reduces accuracy on hard tasks but does not recover diversity. This proves the collapse is embedded in the learned weights during training, not imposed by the CoT generation format.
Task-Dependent Trade-offs: The paper introduces a decomposition showing that diversity loss is a mix of "good" filtering (removing errors) and "bad" homogenization (narrowing correct answers).
- Code/Math: Much of the collapse is beneficial (filtering wrong answers).
- Creative/Value Tasks: The collapse is largely genuine homogenization, risking the imposition of a single perspective.

4. Key Results

A. Lineage-Dependent Collapse

SFT Asymmetry: The "Think" lineage loses 62% of base diversity at SFT, whereas "Instruct" loses only 38%. This is attributed to the "completion homogeneity" of the two-teacher distillation in Think vs. the multi-source variety in Instruct.
DPO Asymmetry: DPO causes a larger drop in "Instruct" (23%) than in "Think" (4%), because "Think" had already collapsed at SFT, leaving little variance for DPO to remove.
RL Reversal: The RL stage slightly increases semantic diversity in "Think" (exploring the diversity floor) but continues to compress "Instruct" (which still had residual variance).
RL-Zero Performance: Models trained via RL-Zero (bypassing SFT/DPO) retain ≥71% of base diversity, confirming that SFT and DPO are the primary drivers of collapse.

B. The Role of CoT

Accuracy vs. Diversity: Removing CoT at inference significantly drops accuracy on hard tasks (e.g., -32% on HumanEval, -28% on MATH-Algebra) but fails to recover diversity. The SBERT diversity scores for "Think" and "Think w/o CoT" remain nearly identical.
Conclusion: The model has already converged to a narrow answer distribution during training; the CoT format is a mechanism for reliability, not the cause of the diversity bottleneck.

C. Quality-Filtered Diversity

Decomposition: On verifiable tasks (e.g., GSM8K, HumanEval), the authors separated diversity loss into:
- Quality-Control: Removal of incorrect outputs.
- Residual: Narrowing of correct outputs.
Findings:
- On Code tasks, ~90% of the collapse is "genuine narrowing" among correct answers (limiting pass@k scaling).
- On Math tasks, ~57–64% is genuine narrowing.
- On IFEval, ~83% is genuine narrowing.
- Impact on Scaling: Models with collapsed correct-answer diversity (Think/Instruct) gain almost nothing from majority voting (e.g., +0.4% on GSM8K), whereas Base and RL-Zero models gain significantly (+24% and +22–26% respectively) because they generate diverse correct solutions.

D. Task-Specific Impacts

Value Pluralism: "Think" models suffer the steepest collapse (-78% to -79%) on tasks like PRISM and TruthfulQA, failing to represent diverse perspectives.
Creative Writing: Both lineages collapse to ~0.20 SBERT, producing formulaic stories despite high quality scores.

5. Significance and Implications

Training Data is Paramount: The study shifts the blame for diversity collapse from the algorithms (SFT/DPO/RL) to the data composition. Narrow data sources (e.g., single/dual-teacher distillation) cause immediate collapse at SFT, while broader data delays it to DPO.
Inference-Time Fixes are Insufficient: Changing generation parameters (temperature) or suppressing CoT cannot recover diversity lost during training. The "diversity floor" is set by the training distribution.
Practical Recommendations:
1. For SFT: Use diverse, multi-source data (more teachers, varied styles) to delay collapse.
2. For RL: Use RL without KL penalties (like RL-Zero) to partially recover diversity, though this may come at a quality cost.
3. For Scaling: Inference-time scaling (e.g., pass@k) is ineffective for post-trained models on tasks where the "correct answer" distribution has collapsed.
Future Directions: The paper calls for targeted interventions in preference optimization objectives and further research into "representational diversity" (preserving specific viewpoints) rather than just distributional spread.

In summary, the paper provides a rigorous causal analysis showing that output diversity collapse is determined during training by data composition, not by the inference format or the specific post-training algorithm alone.