💬 NLP

Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Models

The paper introduces Diverse-NS, a length-controlled data selection strategy that mitigates the systematic bias toward shorter outputs in language models, thereby significantly enhancing response diversity and expressiveness across various creative tasks while maintaining quality.

Original authors: Vijeta Deshpande, Debasmita Ghose, John D. Patterson, Roger Beaty, Anna Rumshisky

Published 2026-02-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vijeta Deshpande, Debasmita Ghose, John D. Patterson, Roger Beaty, Anna Rumshisky

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Short and Boring" Trap

Imagine you have a very talented writer (a Large Language Model) who has been trained to be helpful, safe, and polite. You ask them to write a story.

The problem is that in trying to be "perfect" and safe, the writer has started acting like a nervous student taking a test. They are afraid to write too much because they think shorter answers are better answers.

Why? Because the tools used to grade their work (diversity metrics) are broken. These tools are like a judge who thinks a 3-word sentence is more "creative" than a 30-word paragraph simply because it's shorter. The writer notices this and starts giving you short, safe, repetitive answers to get a good grade.

The authors call this the "Brevity Bias." It's like a chef who stops adding spices because the taste-tester only likes plain food. The result is that the AI becomes less creative, less expressive, and eventually, all AI outputs start sounding exactly the same (a phenomenon called "model collapse").

The Solution: "Diverse, not Short" (Diverse-NS)

The authors created a new strategy called Diverse, not Short (Diverse-NS). Think of this as a new training program for the writer that fixes the broken grading system.

Here is how it works, step-by-step:

The "Two-Draft" Game:
The researchers ask the AI to write a story twice.
- Draft A: The AI writes naturally (its default, often boring style).
- Draft B: The AI is asked to rewrite the story with a specific rule: "Make it more interesting and varied, but keep it the exact same length as Draft A."
The "Fair Judge" Filter:
Usually, if you ask for more creativity, the text gets longer. But the researchers have a strict rule: If Draft B is longer than Draft A, throw it away.
They only keep the pairs where the "more creative" version is roughly the same length as the "boring" version. This teaches the AI: "You can be creative without being wordy."
The "Small Teacher" Trick:
The researchers found something surprising. A smaller, cheaper AI model (like a 7-billion parameter model) can act as a "diversity teacher" for a much larger, more expensive model (like a 13-billion parameter model).
- Analogy: Imagine a small, scrappy local artist teaching a famous, big-budget studio how to paint more uniquely. The small artist knows how to break the rules creatively, and the big studio learns from them. This saves a lot of money and computing power.

The Results: More Flavor, Same Size

The researchers tested this on four different creative tasks:

Divergent Associations: Listing words that are very different from each other.
Persona Generation: Creating unique character profiles (names, jobs, cities).
Alternate Uses: Thinking of weird ways to use a common object (like a brick).
Creative Writing: Writing short stories using three random words.

What happened?

More Variety: The models trained with "Diverse, not Short" produced much more varied and interesting outputs. They used more unique words and ideas.
No Loss in Quality: Usually, when you make things more diverse, they get worse quality (like a messy, chaotic story). But here, the quality stayed high. In some cases, it even got better.
Efficiency: They only needed 3,000 examples to teach the model this new skill. That's like teaching a student for a few hours instead of years.

The New Tool: The "Diversity Decile"

The authors also realized that the old tools for measuring creativity were unfair to long answers. So, they invented a new ruler called Diversity Decile.

The Analogy: Imagine a race. The old ruler just measured how fast you ran, but it ignored that some runners had heavy backpacks (long text) and some didn't (short text).
The New Ruler: The "Diversity Decile" looks at how fast you ran relative to other runners with the same size backpack. It asks: "Is this long story more creative than other long stories?" This ensures that long, expressive answers get the credit they deserve.

Summary

The paper argues that AI models have become too short and repetitive because the way we measure "creativity" accidentally rewards brevity.

By using a strategy called Diverse, not Short, the authors taught models to be creative without making their answers longer. They did this by:

Comparing a boring draft to a creative draft of the same length.
Using a smaller, cheaper AI to teach a bigger AI how to be diverse.
Creating a new, fairer ruler to measure creativity that doesn't penalize long answers.

The result is AI that is more expressive, more creative, and still high-quality, all while being cheaper and faster to train.

Technical Summary: Diverse, not Short (Diverse-NS)

Problem Statement

While alignment techniques (instruction tuning and preference optimization) have made Large Language Models (LLMs) safer and more helpful, they often inadvertently reduce the diversity of model outputs. This reduction in diversity hinders creative ideation, increases output homogeneity, and can lead to "model collapse" when models are repeatedly trained on their own aligned outputs.

A critical, often overlooked confounding factor in this issue is text length. The paper argues that standard diversity metrics (e.g., entropy, Type-Token Ratio) and even reward models used for preference optimization (like ArmoRM) exhibit a systematic bias toward shorter outputs. Because these metrics often assign higher scores to shorter passages, optimizing for them encourages models to be concise rather than expressive. This "brevity bias" undermines the very creativity these models are meant to cultivate, particularly in open-ended tasks where depth and nuance are required.

Methodology: Diverse, not Short (Diverse-NS)

The authors propose Diverse, not Short (Diverse-NS), a length-controlled data selection strategy designed to improve response diversity while maintaining length parity between preferred and rejected responses. The framework operates within a self-learning loop involving three main stages:

1. Data Generation

The authors utilize a creative writing task (generating 5-sentence stories containing three specific words) to elicit diverse responses.

Sequential Prompting: Instead of a single inference, the process uses two steps. First, the model generates a natural response ( $r_1$ ). Second, the model is prompted to redraft the story with additional constraints aimed at reducing repetition and increasing diversity, generating a second response ( $r_2$ ).
Scale: This process generates 200,000 tuples of (prompt, $r_1$ , $r_2$ ) for both LLaMA-3.1-8B and Olmo-2-7B.

2. Data Filtration (The Core Innovation)

To create high-quality preference pairs for Direct Preference Optimization (DPO), the authors filter the generated data based on four strict criteria applied in order:

Quality Threshold: The quality of $r_2$ must be greater than or equal to the 50th percentile of all $r_1$ quality scores.
Quality Improvement: The quality of $r_2$ must be strictly greater than $r_1$ .
Diversity Improvement: The diversity of $r_2$ must be strictly greater than $r_1$ .
Length Parity: The absolute difference in word count between $r_1$ and $r_2$ must be at most five words.

This length constraint is crucial to prevent the selection of shorter, "easier" diverse responses.

Metrics: The primary method uses Entropy for diversity and ArmoRM scores for quality.
Diverse-NS-Lite: To reduce computational cost (as calculating entropy and ArmoRM requires multiple inference passes), the authors introduce a lightweight variant. This replaces Entropy with Type-Token Ratio (TTR) and ArmoRM with the MAAS index, which showed the highest correlation with the original metrics in their analysis.

3. Preference Tuning

The filtered dataset is limited to 3,000 preference pairs (selected by sorting on entropy gain) to fine-tune the base model using Direct Preference Optimization (DPO). The authors also explore "small-to-large" transfer, using data generated by a smaller model (Olmo-7B) to train a larger model (Olmo-13B).

4. Length-Adjusted Evaluation

To address the evaluation bias, the authors introduce $\Delta$ Diversity Decile ( $\Delta$ DD).

They build a decile map based on 800,000 stories, grouping responses by word count and calculating diversity thresholds for each length.
During evaluation, a new response is assigned a decile rank based on its length and diversity score relative to the map.
$\Delta$ DD is calculated as the difference between the tuned model's decile score and the base model's, ensuring diversity gains are not simply artifacts of increased length.

Key Contributions

Diverse-NS Strategy: A data selection method that significantly improves response diversity for LLaMA-3.1-8B and Olmo-2-7B using only 3,000 preference pairs, explicitly controlling for length bias.
Diverse-NS-Lite: A computationally efficient variant using proxy metrics (TTR and MAAS) that achieves comparable performance to the full method while drastically reducing filtering costs.
Small-to-Large Transfer: Evidence that smaller models (e.g., Olmo-2-7B) can serve as effective "diversity teachers" for larger models (e.g., Olmo-2-13B), enabling low-cost diversity alignment.
Length-Controlled Metric: The introduction of Diversity Decile, a metric that adjusts diversity scores for text length to prevent brevity bias in evaluation.
Dataset Release: A high-quality dataset of 6,000 preference pairs generated from LLaMA-3.1-8B and Olmo-2-7B.

Results

The authors evaluated the method on four creative generation tasks: Divergent Associations (DAT), Persona Generation (PGT), Alternate Uses (AUT), and Creative Writing (CWT).

Diversity Gains: Both Diverse-NS and Diverse-NS-Lite consistently outperformed the base models and the DivPO baseline (a recent method that does not control for length) across all tasks and model sizes.
- In the DAT and PGT tasks, the methods produced significantly more unique words, names, cities, and occupations.
- In CWT, Diverse-NS achieved the highest Divergent Semantic Integration (DSI) scores and 4-gram diversity.
Statistical Significance: Independent t-tests confirmed that D-NS and D-NS-Lite produce significantly more diverse responses than the base model and DivPO (p < 0.001 for D-NS vs. Base). The difference between D-NS and D-NS-Lite was not statistically significant, validating the efficiency of the Lite version.
Quality Preservation: While diversity and quality often trade off, Diverse-NS maintained or slightly improved quality (measured by ArmoRM) in several cases (e.g., Olmo-7B and Olmo-13B). In cases where quality dipped slightly, the absolute diversity values still exceeded the base model, and human evaluation confirmed that the diversity gains were perceptible and preferred.
Human Evaluation: In pairwise comparisons on CWT, human annotators preferred D-NS over the base model and DivPO in the majority of cases (win rates up to 72.73% against DivPO).

Significance and Claims

The paper claims that by explicitly addressing the length bias inherent in both training data selection and diversity metrics, Diverse-NS offers a practical and scalable solution for boosting diversity in aligned LLMs without sacrificing quality.

Efficiency: The method achieves substantial improvements with a very small dataset (3k pairs) and minimal training time (under two hours on a single GPU).
Accessibility: By demonstrating that smaller models can act as diversity teachers and by offering a lightweight filtering variant, the work lowers the resource barrier for diversity-aware alignment.
Foundational Shift: The authors argue that previous failures to improve diversity may stem from the unaddressed brevity bias in metrics. Their work sets a foundation for more expressive language generation by decoupling diversity from brevity.

The authors modestly note limitations, including reliance on a single diversity metric for filtering, focus on a single task (story writing), and the investigation of only a single round of self-training. They suggest future work should explore multi-metric optimization and multi-round self-learning.