💬 NLP

Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores

This paper introduces "diversity," an open-source Python package and interactive platform that standardizes the measurement of text diversity in LLM outputs by empirically validating that a combination of low-correlation metrics—including compression ratios, long n-gram self-repetition, Self-BLEU, and BERTScore—effectively captures text repetition patterns.

Original authors: Chantal Shaib, Venkata S. Govindarajan, Joe Barrow, Jiuding Sun, Alexa F. Siu, Byron C. Wallace, Ani Nenkova

Published 2026-02-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chantal Shaib, Venkata S. Govindarajan, Joe Barrow, Jiuding Sun, Alexa F. Siu, Byron C. Wallace, Ani Nenkova

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef running a massive kitchen. You have hired six different robots (AI models) to cook meals based on customer orders. Your job is to check if these robots are actually cooking unique dishes or if they are just copying the same recipe over and over again, perhaps just changing the garnish slightly.

This paper is about building a universal ruler to measure how "boring" or "repetitive" these robot chefs are, and then handing that ruler to everyone else so they can use it too.

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Copy-Paste" Kitchen

When Large Language Models (LLMs) write text, they sometimes get stuck in a loop. They might say, "The weather is nice," then "The weather is nice," then "The weather is nice" in different documents.

The Issue: Until now, there was no standard way to measure this. Some chefs might use a ruler that measures the length of the sentence, while others measure the color of the words. It's like trying to compare a marathon runner's speed to a swimmer's speed using different units.
The Consequence: If a model is just repeating "boilerplate" (canned) phrases, it feels robotic and low-quality, but we couldn't easily prove it across huge batches of text.

2. The Solution: The "Diversity" Toolkit

The authors built a new tool called diversity. Think of this as a smart kitchen scale that doesn't just weigh food; it analyzes the ingredients.

The Package: It's a free software tool (like a set of measuring cups) that anyone can download to check their text.
The Website: They also built a visual playground (a "dashboard") where you can upload your text and see exactly where the robots are repeating themselves. It highlights the boring parts in red, like a teacher marking up a student's essay for using too many clichés.

3. The Secret Sauce: Compression is Key

The paper discovered a clever trick to measure repetition without doing heavy math.

The Analogy: Imagine you have a 100-page story where the same paragraph is repeated 50 times. If you try to zip that file up (compress it) to save space, the computer will say, "Oh, this is easy! I can shrink this huge file down to just 5 pages because it's all the same stuff."
The Finding: The authors found that Compression Ratio (how much a file shrinks when zipped) is a super-fast, super-accurate way to measure repetition.
- High Compression = The text is very repetitive (boring).
- Low Compression = The text is unique and diverse (interesting).
Why it matters: Other methods require the computer to read every single word and compare it to every other word (like reading a library to find duplicates). Compression is like just looking at the file size; it's instant and cheap.

4. The "Length" Trap

The paper warns about a major pitfall: Text Length.

The Analogy: If you ask Robot A to write a 10-word sentence and Robot B to write a 100-word essay, Robot B will naturally have more unique words just because it's longer. If you compare them directly, Robot B looks "more diverse" simply because it talked more.
The Fix: You have to compare apples to apples. If you want to know which robot is actually more creative, you have to make sure they are writing the same amount of text, or you have to adjust your math to account for the length. The paper emphasizes that you can't trust diversity scores unless you know how long the text is.

5. The Results: Who is the Best Chef?

The authors tested six famous AI models (like Llama-2, GPT-4, Mistral) on news summarization tasks.

The Surprise: Some models that wrote very long summaries were actually the least diverse. They were just rambling and repeating themselves to fill space.
The Winners: Models like GPT-4 tended to be more concise and diverse.
The Human Comparison: Interestingly, when they compared the robots to human journalists, the robots often sounded more repetitive. However, some specific metrics (like counting how often a model repeats a 4-word phrase) were better at spotting this than others.

6. The Takeaway

The authors are saying: "Stop guessing. Use our ruler."
They recommend a combination of three simple checks to get a true picture of text diversity:

Compression Ratio: How much does the text shrink when zipped? (Fast & Easy).
Self-Repetition: Does the model repeat long phrases?
Self-BLEU: How similar are the robot's different answers to each other?

By using these tools, researchers and developers can stop relying on gut feelings and start using data to ensure their AI models are actually being creative, not just being lazy copycats.

In short: The paper gives us a free, easy-to-use toolkit to spot when AI is being boring, using the simple logic that "if you can zip it up easily, it's probably repetitive."

1. Problem Statement

Large Language Models (LLMs) are increasingly evaluated on accuracy, factuality, and coherence, but text diversity (the extent to which outputs avoid repetitive "boilerplate" or canned responses) lacks standardized measurement.

Current Gaps: There is no consensus on which metrics to use, how to visualize repetition across large corpora, or how to handle the confounding factor of text length (longer texts naturally contain more repetition, skewing metrics).
Human Limitations: Human judgments of diversity are difficult to collect reliably and often conflate quality with diversity.
Computational Cost: Many existing metrics (e.g., pairwise similarity scores) are computationally expensive, making them infeasible for large-scale datasets.

2. Methodology

The authors propose a holistic approach combining a new software tool, a comparative analysis of existing metrics, and an investigation into text length as a confounder.

A. The `diversity` Package and Platform

The authors released diversity, an open-source Python package and a corresponding web-based UI. The tool supports:

Metric Calculation: Computation of lexical, syntactic, and semantic diversity scores.
Visualization: Interactive highlighting of repetitive Part-of-Speech (PoS) patterns and exact text matches.
Efficiency: Implementation of memoization and batch processing to handle large datasets.

B. Metrics Evaluated

The study empirically investigates the convergent validity of several existing metrics across English texts:

Homogenization Scores:
- Self-BLEU: Measures similarity between all pairs of generated texts using BLEU.
- ROUGE-L / BERTScore: Aggregate similarity across pairs (using longest common subsequence or semantic embeddings).
Repetition Scores:
- Self-Repetition Score (SRS): Measures the frequency of long n-grams ( $n \ge 4$ ) appearing across different outputs.
Token/Type Ratios:
- Moving Average Token-Type Ratio (MATTR): Unique tokens divided by total tokens in moving windows.
- N-Gram Diversity Score (NGD): Ratio of unique n-grams to total n-grams.
Compression Ratios (CR):
- CR: The ratio of the compressed size (using gzip) of concatenated texts to the original size. High CR implies high redundancy.
- CR:POS: Compression ratio applied specifically to sequences of Part-of-Speech tags to capture syntactic repetition.

C. Experimental Setup

Datasets: CNN/DailyMail and XSUM (news summarization).
Models: Six instruction-tuned LLMs (Llama-2, GPT-4, FlanT5-XXL, StableLM, Mistral, StableBeluga) and human-written baselines (articles, reference summaries).
Control for Length: The authors analyzed the impact of text length and tested a "truncation" method (cutting all texts to the length of the shortest source) to isolate diversity from length effects.

3. Key Results

A. Metric Correlations and Redundancy

Compression Ratio (CR) is highly correlated with most n-gram based diversity scores (e.g., NGD, MATTR) but is significantly faster to compute.
Self-BLEU and Homogenization (BERTScore) show weak correlation with CR and other n-gram metrics, suggesting they capture different information (semantic vs. lexical).
Recommendation: A combination of CR (or CR:POS), Self-Repetition, and Self-BLEU is sufficient. They offer low mutual correlation, covering different aspects of diversity without redundant computation.

B. The Confounder of Text Length

Text length is a major confounder. Models generating longer summaries (e.g., StableLM) often appear less diverse simply because longer texts have more opportunities for repetition.
Truncation Analysis: When texts are truncated to a uniform length:
- Model rankings change significantly.
- Self-BLEU and CR consistently indicate that model-generated text is less diverse than human text.
- Homogenization (BERTScore) barely varies across sources when length is controlled, rendering it less useful for distinguishing diversity in summarization tasks.

C. Specific Metric Performance

CR:POS (Part-of-Speech Compression): Successfully identifies differences between human and model-generated text that other metrics miss. It captures syntactic template repetition.
Self-Repetition: The only metric that occasionally flagged model text as more diverse than human text on CNN/DM, likely due to the specific nature of long n-gram repetition in human news writing vs. LLM generation.
BERTScore: Found to be limited; it failed to show significant differences between human and model texts in this context.

D. Efficiency

CR is orders of magnitude faster than pairwise similarity scores (Self-BLEU, BERTScore). For large datasets (e.g., 50k instructions), computing Self-BLEU or BERTScore could take hundreds of hours, whereas CR is nearly instantaneous.

4. Key Contributions

Tool Release: The diversity Python package and a web UI for interactive exploration of text repetition and diversity metrics.
Standardization: A comparative analysis establishing that Compression Ratios are a fast, effective proxy for complex n-gram metrics.
Methodological Insight: A demonstration that text length must be controlled for (via reporting or truncation) to draw valid conclusions about diversity.
Metric Recommendations: Proposing a minimal, non-redundant set of metrics for reporting: CR (or CR:POS), Self-Repetition, and Self-BLEU.

5. Significance

This work addresses a critical gap in LLM evaluation by providing a standardized, efficient, and open-source framework for measuring text diversity.

Practical Utility: Researchers can now easily visualize "boilerplate" content and compare model outputs without prohibitive computational costs.
Scientific Rigor: By highlighting the confounding effect of text length, the paper prevents erroneous conclusions about model performance (e.g., assuming a model is less diverse simply because it writes longer).
Future Direction: The findings suggest that future evaluations should prioritize fast, length-aware metrics like Compression Ratios over expensive pairwise semantic similarity scores for large-scale diversity analysis.