💬 NLP

Evaluating the Diversity and Quality of LLM Generated Content

This paper introduces a framework for measuring "effective semantic diversity" to demonstrate that while preference-tuned LLMs may appear less diverse under standard metrics, they actually generate a greater variety of high-quality outputs compared to base or supervised fine-tuned models, with smaller models proving more parameter-efficient for producing unique content.

Original authors: Alexander Shypula, Shuo Li, Botong Zhang, Vishakh Padmakumar, Kayo Yin, Osbert Bastani

Published 2026-02-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alexander Shypula, Shuo Li, Botong Zhang, Vishakh Padmakumar, Kayo Yin, Osbert Bastani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical chef (the AI) who can cook up millions of different dishes. You want this chef to be creative (making many unique dishes) but also delicious (the dishes must actually taste good and be safe to eat).

For a long time, people worried that if you trained this chef to always make "perfect" dishes according to human taste, they would become boring robots, only making the same three safe dishes over and over again. They feared the chef would lose their "spark."

This paper, presented at a major AI conference, asks a simple but profound question: Does making an AI "smarter" and "safer" actually kill its creativity, or does it just make its creativity more useful?

Here is the breakdown of their findings, using some kitchen metaphors.

1. The Trap of "Fake Diversity"

The authors point out a common mistake in how we measure creativity.

The Old Way: Imagine you ask the chef to make 100 dishes. If the chef makes 100 dishes that are all just "burnt toast" but with slightly different shades of brown, a simple counter might say, "Wow! 100 unique dishes!"
The Problem: While they look different on the outside, they are all inedible. This is diversity without quality. It's like a library full of books where every page is just random scribbles. It's diverse, but useless.

2. The New Metric: "Effective Semantic Diversity"

The authors introduce a new way to measure creativity called Effective Semantic Diversity.

The Analogy: Instead of just counting how many dishes the chef made, we first throw away all the burnt toast and raw eggs. We only look at the dishes that are actually edible (valid). Then, we ask: "How many different delicious meals did we get?"
The Result: This measures the useful variety. It's the difference between a chef who makes 1,000 variations of burnt toast (high raw diversity, low value) and a chef who makes 50 distinct, gourmet meals (lower raw count, but high effective diversity).

3. The Big Surprise: "Good" AI is Actually More Creative

The researchers tested this on different types of AI chefs:

The Base Chef: The raw, untrained model.
The "Trained" Chefs: Models that have been fine-tuned using advanced techniques (like RLHF, DPO, or PPO) to follow instructions and be helpful.

The Counter-Intuitive Finding:
When they looked at the "raw" output (ignoring quality), the trained chefs seemed less diverse. They were more focused and less chaotic.
However, when they applied their new "Effective Diversity" filter (only counting the good stuff):

The Trained Chefs produced more unique, high-quality ideas than the raw chefs.
Why? Because the raw chefs spent 90% of their time making "burnt toast" (garbage). The trained chefs spent almost all their time making "gourmet meals." Even if the trained chefs made fewer total dishes, the number of good, unique dishes was much higher.

The Takeaway: Making an AI "safer" and "better" doesn't kill its creativity; it actually unlocks more useful creativity by stopping it from wasting time on nonsense.

4. The Size vs. Efficiency Puzzle

The paper also looked at the size of the chefs (the number of parameters in the model).

The Big Chefs (70 Billion parameters): They are like master chefs with a massive kitchen. They can cook up a huge variety of unique, high-quality meals.
The Small Chefs (8 Billion or even 500 Million parameters): They are like compact food trucks.
The Finding: If you have a limited budget (a fixed amount of time or money to generate ideas), the small chefs are actually more efficient.
- Analogy: If you need 1,000 unique recipes for a party, hiring one giant, expensive master chef to cook 1,000 times might be overkill and slow. Instead, hiring 10 small, agile food trucks to each cook 100 times might get you more unique results for the same cost.

5. Code vs. Creative Writing

The researchers tested this on two types of tasks:

Coding (Math/Logic): Here, "quality" is easy to check. Does the code run without crashing? The trained chefs were much better at making code that actually worked, leading to more unique, working programs.
Creative Writing (Stories): Here, "quality" is subjective. The trained chefs still produced more unique, high-quality stories, but they also tended to use more varied words and styles compared to the raw chefs.

Summary

The paper argues that we shouldn't fear "alignment" (training AI to be helpful).

Old Fear: "If we train AI to be good, it will become a boring clone."
New Reality: "If we train AI to be good, it stops wasting time on garbage and starts producing a much richer variety of actually useful ideas."

In short: A chaotic mess of 1,000 bad ideas is less valuable than a curated collection of 50 brilliant ones. The "smart" AI gives us the brilliant ones.

1. Problem Statement

Recent advancements in Large Language Models (LLMs) rely heavily on preference-tuning techniques (e.g., RLHF, PPO, GRPO, DPO) to align models with human preferences. A prevailing narrative suggests that these techniques reduce output diversity, creating a dilemma for applications requiring varied outputs (e.g., creative writing, synthetic data generation).

However, the authors argue that diversity without quality is practically useless. Random tokens are maximally diverse but meaningless. Existing evaluation metrics often fail to account for the trade-off between quality and diversity, leading to misleading conclusions where base models appear more diverse simply because they generate more invalid or low-quality outputs. The core problem is the lack of a framework to measure Effective Semantic Diversity: diversity among outputs that meet a specific threshold of quality.

2. Methodology

A. Framework: Effective Semantic Diversity

The authors propose a principled framework that jointly measures quality and diversity without requiring human intervention at inference time.

Validity Function ( $V$ ): Determines if a generation is high-quality/valid.
- For Code: A program is valid if it executes without errors and produces non-null outputs on a suite of test cases.
- For Natural Language: An LLM judge evaluates quality based on criteria (e.g., coherence, relevance) and normalizes the score.
Semantic Function ( $S$ ): Maps a generation to its semantic meaning, ignoring superficial lexical differences.
- For Code: Two programs are semantically equivalent if they produce identical outputs across all test cases.
- For Natural Language: An LLM judge evaluates conceptual/thematic overlap.
Metrics:
1. Fixed Semantic Diversity ( $Div_{fixed}$ ): The proportion of generations that are both valid and semantically unique.
2. Pairwise Semantic Diversity ( $Div_{pair}$ ): A pairwise distance metric normalized by the total number of pairs. This is robust to variations in the number of valid samples across prompts.
  $d_{sem}(g_j, g_k) = \begin{cases} 0 & \text{if } V(g_j)=0 \text{ or } V(g_k)=0 \\ 0 & \text{if } V=1 \text{ and } S(g_j)=S(g_k) \\ 1 & \text{otherwise} \end{cases}$

B. Experimental Setup

Datasets:
- Code: 108 open-ended programming tasks adapted from CodeNet/AlphaCode. These require generating functions that solve abstract problems.
- Natural Language: 100 creative writing prompts from WritingPrompts.
Models: A wide range of LLMs including LLaMA 2/3.1 (7B, 70B) and Tulu variants (Base, SFT, DPO, PPO, GRPO).
Comparisons: The study isolates the effects of:
- Post-training methods (Base vs. SFT vs. DPO vs. RL/PPO/GRPO).
- Model size (Small vs. Large).
- Parameter efficiency (Diversity per parameter).
Baselines: The authors compare their metric against standard lexical (Distinct-N), syntactic (AST-based), and neural (Cosine similarity via CodeBERTScore) diversity metrics.

3. Key Contributions

Introduction of Effective Semantic Diversity: A novel metric that penalizes low-quality outputs, arguing that diversity is only valuable when the content is usable.
Automated Evaluation Pipeline: A framework that uses code execution and LLM judges to evaluate diversity and quality without human annotators, avoiding the "LLM judging LLM" bias by using execution traces for code.
Counterintuitive Findings on Preference Tuning: Challenging the consensus that preference tuning reduces diversity.
Parameter Efficiency Analysis: Investigating which model sizes are most efficient for generating unique content within a fixed sampling budget.

4. Key Results

A. Preference Tuning Increases Effective Diversity

Main Finding: While preference-tuned models (RL/DPO) often show lower raw lexical or neural diversity compared to base models, they exhibit higher Effective Semantic Diversity.
Mechanism: Preference tuning drastically increases the proportion of valid/high-quality outputs. Even if the rate of unique ideas per valid sample drops slightly, the sheer volume of valid samples generated by tuned models results in a higher total count of unique, high-quality ideas.
Code vs. Text:
- Code: RL-tuned models reduce lexical and syntactic diversity (more standard solutions) but maintain or improve semantic diversity due to higher validity.
- Creative Writing: Preference tuning actually increases lexical diversity (more varied word choices) alongside quality.

B. The "Diversity Collapse" Misconception

When using metrics that ignore quality (e.g., raw cosine distance), base models often appear most diverse because they generate a high volume of incoherent, invalid outputs.
Once quality is factored in, the "diversity collapse" narrative is reversed: RL-tuned models are the most diverse in terms of useful, unique outputs.

C. Model Size and Efficiency

Larger Models: Exhibit higher absolute effective semantic diversity than smaller models.
Parameter Efficiency: When measuring diversity per parameter (efficiency) within a fixed sampling budget (e.g., 32 generations), smaller models (down to ~500M parameters) are often more efficient at generating unique, valid content than massive 70B models. This suggests smaller models are optimal for synthetic data generation tasks where cost is a constraint.

D. Impact of Sampling Temperature

The framework reveals that both extremely low and extremely high temperatures reduce effective diversity.
Low temperature leads to repetition (low diversity).
High temperature leads to incoherence (low validity), which the metric penalizes.
Preference-tuned models often find a "sweet spot" at moderate temperatures that maximizes effective diversity.

5. Significance and Implications

Redefining Evaluation: The paper argues that the community must shift from measuring "raw diversity" to "effective semantic diversity" to accurately assess LLM utility in open-ended tasks.
Synthetic Data Generation: For tasks like generating training data, the findings suggest that using smaller, preference-tuned models may be more cost-effective than using massive base models, as they generate a higher density of unique, valid samples per compute unit.
Creative Assistance: Preference tuning (RLHF) does not stifle creativity; rather, it enhances the utility of creative outputs by ensuring they are coherent and relevant while maintaining a broad range of ideas.
Methodological Shift: The use of program execution as a ground-truth validity check provides a robust, low-cost alternative to human evaluation for diversity studies, setting a precedent for future automated evaluation benchmarks.

In conclusion, the paper demonstrates that quality and diversity are not zero-sum. Preference tuning techniques, often criticized for reducing diversity, actually enhance the practical diversity of LLMs by filtering out noise and ensuring that the generated variety is meaningful and usable.