💬 NLP

Task-Dependent Evaluation of LLM Output Homogenization: A Taxonomy-Guided Framework

This paper proposes a task-dependent framework for evaluating LLM output homogenization, introducing a taxonomy of functional diversity validated by user studies to demonstrate that diversity and quality are not inherently trade-offs when diversity is conceptualized according to specific task requirements.

Original authors: Shomik Jain, Jack Lanchantin, Maximilian Nickel, Candace Ross, Karen Ullrich, Ashia Wilson, Jamelle Watson-Daniels

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shomik Jain, Jack Lanchantin, Maximilian Nickel, Candace Ross, Karen Ullrich, Ashia Wilson, Jamelle Watson-Daniels

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read friend who can answer any question you ask. Sometimes, you ask them for a creative story, and they tell you the exact same plot every time, just changing a few words. Other times, you ask them a math problem, and they give you the right answer but explain it in a boring, repetitive way.

This paper is about fixing that "stuck record" feeling. The authors call it homogenization—when AI models start sounding like they're all reading from the same script.

Here is the core idea, broken down with some everyday analogies:

1. The Problem: One Size Does Not Fit All

The paper argues that we've been judging AI diversity wrong. We've been using a single ruler to measure everything, like trying to measure the "spiciness" of a soup and the "color" of a painting with the same tool.

The Creative Task (The Jazz Musician): If you ask an AI to write a poem about time, you want it to be like a jazz musician improvising. You want different melodies, different rhythms, and different emotions. If it just says "Time is a river" three times in a row, that's boring. That's bad homogenization.
The Factual Task (The GPS): If you ask an AI, "What is the capital of France?", you want it to be like a GPS. You don't want it to say "Paris" once, then "Paris (sort of)," then "The city of lights." You want it to be consistent and accurate. If it tries to be "creative" and says "Maybe Paris, maybe London," that's actually a failure.

The Big Mistake: Previous research tried to force all AI answers to be different, even when they shouldn't be. It's like telling a GPS to give you three different routes to the same destination every time you ask. That's confusing and unhelpful.

2. The Solution: The "Task Taxonomy" (The Menu)

The authors created a menu (a taxonomy) of 8 different types of tasks. Think of this as a restaurant menu where the chef knows exactly how to cook each dish differently.

Category A (The Math Problem): "Give me the one correct answer." (Consistency is key).
Category G (The Creative Writing): "Give me a story with a unique plot." (Variety is key).
Category H (The Opinion): "What's a good gift for Mom?" (Different perspectives are key).

By sorting the user's question into the right "dish" on the menu, the AI knows whether to be a strict librarian or a wild artist.

3. The Fix: "Task-Dependent Sampling" (The Smart Switch)

The paper introduces a new way to talk to the AI called Task-Dependent Sampling.

Imagine the AI is a chef.

Old Way: You tell the chef, "Make me 5 different meals!" The chef panics. For a math problem, they might give you 5 different wrong answers just to look "diverse." For a creative story, they might just change the font color but keep the story the same.
New Way: You tell the chef, "This is a Math problem. Give me 5 different ways to solve it, but make sure they all lead to the same answer." Then, you say, "This is a Story prompt. Give me 5 totally different stories with different characters and endings."

The AI now knows what kind of difference you want.

For math, it varies the strategy (the steps), not the answer.
For stories, it varies the plot and tone.

4. The Results: Breaking the "Quality vs. Variety" Myth

There was a common belief that you have to choose: either the AI is smart and consistent (high quality) OR it is creative and varied (high diversity). You couldn't have both.

The paper proves this is a myth. It's like saying you can't have a car that is both safe and fast.

When you measure "quality" correctly (does the math answer work? is the story engaging?), the AI can be both high-quality and highly diverse.
The "trade-off" only existed because we were using the wrong measuring sticks (like counting how many different words were used, rather than checking if the ideas were actually different).

5. Why This Matters

If we don't fix this, AI becomes dangerous in two ways:

The Boredom Trap: You ask for a story, and it gives you the same "hero saves the day" plot a million times.
The Hallucination Trap: You ask for a fact, and the AI tries to be "creative" by inventing fake facts just to sound different.

In a nutshell:
This paper teaches us to stop treating the AI like a broken record that needs to be shaken up. Instead, we should treat it like a skilled employee who knows when to be a strict accountant (for facts) and when to be a wild artist (for creativity). By giving the AI the right instructions for the specific job, we get better answers that are both accurate and refreshingly unique.

1. Problem Statement

Large Language Models (LLMs) frequently exhibit output homogenization, where they generate repetitive or similar responses to different prompts. While this is often viewed as a negative trait, the paper argues that the desirability of diversity is task-dependent:

Undesirable Homogenization: In creative writing or opinion tasks, models often converge on the same tropes (e.g., "time is a river" metaphors) or singular viewpoints, reducing utility and potentially causing representation harms.
Desirable Homogenization: In objective tasks (e.g., math problems, fact retrieval), consistency is crucial. Generating multiple different answers to a single correct math problem is a failure mode, not a feature.

The Gap: Existing benchmarks (e.g., Novelty-Bench, Infinity-Chat) and diversity metrics (vocabulary overlap, embedding cosine distance) apply a task-agnostic notion of diversity. They fail to distinguish between lexical variation (different words) and functional variation (meaningfully different strategies, perspectives, or creative elements). This leads to a perceived "diversity-quality trade-off," where increasing diversity often degrades quality because the metrics are misaligned with the task's actual requirements.

2. Methodology

The authors propose a Task-Dependent Framework consisting of three core components:

A. A Taxonomy of Functional Diversity

The paper introduces an 8-category taxonomy (Table 1) that defines what constitutes "meaningful diversity" for specific task types. The categories span a spectrum of reward verifiability:

Well-Specified Singular Objective: Single verifiable answer (e.g., "Largest Spanish-speaking country?"). Diversity Goal: None (consistency required).
Underspecified Singular Objective: Multiple verifiable answers (e.g., "Name a Spanish-speaking country"). Diversity Goal: Different correct answers.
Random Generation: Randomizing over finite options (e.g., "Roll a die"). Diversity Goal: Different pseudo-random options.
Problem-Solving Objective: Single answer, multiple strategies (e.g., Math proofs). Diversity Goal: Different solution strategies.
Problem-Solving Subjective: Multiple correct answers/strategies (e.g., Design a room). Diversity Goal: Different strategies or solutions.
Encyclopedia Inquiry: Factual information with credible references. Diversity Goal: Different factual perspectives.
Creative Writing: Open-ended expression. Diversity Goal: Different creative elements (plot, tone, genre).
Advice or Opinions: Subjective feedback. Diversity Goal: Different viewpoints or perspectives.

B. Task-Dependent Sampling Technique

The authors propose a two-step inference-time method to mitigate homogenization only where it is undesired:

Classification: The model (or a classifier) determines the prompt's category from the taxonomy.
Augmented Prompting: The system prompt or regeneration prompt is dynamically modified to explicitly request the specific type of functional diversity defined for that category.
- Example: For "Problem-Solving Objective," the prompt instructs: "Generate different solutions, each with a different problem-solving strategy."
- Example: For "Well-Specified Objective," the prompt instructs: "The answer should remain the same; slight variation in wording is allowed."

C. Evaluation Metrics

Instead of relying on general embedding distances, the authors use Taxonomy-Guided LLM-Judges.

An LLM judge is provided with the prompt, two responses, and the specific functional diversity definition for that task category.
The judge determines if the responses are "functionally equivalent" (redundant) or "functionally diverse" (meaningfully different).
This metric is validated against human annotations.

3. Key Contributions

Taxonomy of Functional Diversity: A structured framework operationalizing how diversity should be measured differently across 8 distinct task categories, moving beyond generic "novelty."
Validation of Human Alignment: A user study ( $n=22$ ) confirms that the taxonomy aligns with human perception of functional diversity. Taxonomy-guided LLM-judges correlate strongly with human judgments ( $r_s = 0.93$ ), significantly outperforming embedding-based ($0.81$) and vocabulary-based ($0.67$) metrics.
Task-Dependent Sampling Algorithm: A practical inference-time technique that increases functional diversity in creative/opinion tasks while preserving homogenization in objective tasks.
Debunking the Diversity-Quality Trade-off: The paper demonstrates that the trade-off is an artifact of using task-agnostic metrics. When diversity and quality are measured using task-dependent definitions, the trade-off becomes negligible or non-existent.

4. Key Results

Diversity Gains: Task-dependent sampling significantly increases the number of functionally diverse responses in open-ended tasks (Creative Writing, Advice) compared to default temperature sampling or generic prompting.
- Example: On the "Write a metaphor involving time" prompt, task-dependent sampling increased the convex hull area of response embeddings by 1.7x to 3.5x across GPT-4o, Claude, and Gemini, producing conceptually distinct metaphors (e.g., "Time is a sculptor" vs. "Time is a river") rather than lexical variations of the same metaphor.
Preservation of Consistency: In "Well-Specified Objective" tasks (Category A), the method successfully maintained homogenization (producing ~1 unique answer), whereas generic diversity-promoting methods incorrectly generated multiple different answers.
Quality Maintenance: Using task-dependent grading checklists, the study found that increasing functional diversity did not degrade response quality. In contrast, using general reward models (like Athene-RM) showed a negative correlation, reinforcing the argument that standard reward models penalize necessary diversity in creative tasks.
Cross-Model Homogenization: The method also reduced "Artificial Hivemind" effects (inter-model homogenization) in the Infinity-Chat benchmark, particularly for Creative Writing and Advice tasks, where models previously converged on similar viewpoints.

5. Significance and Implications

Redefining Evaluation: The paper argues that current homogenization benchmarks are flawed because they treat all tasks as if they require the same type of diversity. This framework provides a mechanism to evaluate models more accurately based on the intent of the user.
Safety and Ethics: By distinguishing between tasks where diversity is needed (to avoid representation bias in opinions) and tasks where consistency is needed (to avoid hallucination in facts), the framework offers a safer approach to deploying LLMs.
Mitigation Strategy: It provides a lightweight, inference-time solution (prompt engineering) that does not require retraining models, making it immediately applicable to existing systems.
Future Alignment: The authors suggest that future alignment methods (like RLHF or DPO) should incorporate task-dependent diversity signals to prevent the "collapse" of functional diversity in creative domains while maintaining factual consistency in objective domains.

In summary, the paper shifts the paradigm from "maximizing diversity at all costs" to "optimizing for task-appropriate diversity," demonstrating that this approach improves both the utility and safety of LLM outputs without sacrificing quality.