💬 NLP

Death of the Novel(ty): Beyond n-Gram Novelty as a Metric for Textual Creativity

This paper argues that n-gram novelty is an insufficient standalone metric for textual creativity because it fails to account for appropriateness, demonstrating through extensive human annotation that while high novelty often lacks pragmatic value in AI-generated text, frontier LLMs can be fine-tuned to better identify creative and non-pragmatic expressions than traditional n-gram methods.

Original authors: Arkadiy Saakyan, Najoung Kim, Smaranda Muresan, Tuhin Chakrabarty

Published 2026-03-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Arkadiy Saakyan, Najoung Kim, Smaranda Muresan, Tuhin Chakrabarty

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge at a talent show. For years, the only rule you had for picking the winner was: "The act must be something you've never seen before."

If a magician pulled a rabbit out of a hat, you'd say, "Boring, I've seen that!" But if a magician pulled a dragon out of a hat, you'd scream, "Wow! That's new! That's creative!"

This is exactly how we've been measuring AI writing for a long time. We use a metric called n-gram novelty. It's basically a giant search engine that asks: "Has this exact phrase appeared in the billions of books and websites the AI was trained on?" If the answer is "No," we assume the AI is being creative.

This paper is the talent show judge realizing that rule is broken.

Here is the story of what the researchers found, explained simply:

1. The "Gibberish" Problem

The researchers (a team of writers and computer scientists) asked 26 professional writers to read stories written by humans and by AI. They didn't just look for "newness"; they looked for creativity.

They defined true creativity as a sandwich with two slices of bread:

Slice A (Novelty): It must be original and surprising.
Slice B (Appropriateness): It must make sense and fit the story.

The Shocking Discovery:
The researchers found that the "Newness Rule" (n-gram novelty) is a terrible judge.

91% of the time, when the AI generated text that was "super new" (never seen before), it was actually garbage. It was like a magician pulling a dragon out of a hat, but the dragon was made of spaghetti and was screaming in a language no one understood. It was new, but it wasn't creative; it was just nonsense.
Conversely, some of the most beautiful, creative lines written by humans had low novelty scores. They used common words, but the way they were put together was emotionally perfect. The "Newness Rule" would have rejected these as "boring," even though they were masterpieces.

2. The AI's "Over-Engineered" Trap

The study found a weird pattern in how AI writes.

Humans: When humans try to be creative, they balance new ideas with making sense.
AI: When AI tries to be "novel" (to avoid copying its training data), it starts to break. The more the AI tries to be unique, the more likely it is to write sentences that don't make sense in the context.

The Metaphor:
Imagine a chef who is told, "You must never use an ingredient that has ever been in a kitchen before."

A human chef might say, "Okay, I'll use a rare mushroom I found in the forest." (Delicious and new).
The AI chef, trying to follow the rule strictly, might say, "Okay, I'll put a tire and a toaster in the soup." (Technically new, but inedible).

The paper shows that as AI models get bigger and try harder to be "novel," they accidentally start serving "tire soup."

3. The "Robot Judge" Experiment

The researchers then asked: "Can we teach a super-smart AI (like GPT-5) to be the judge instead of us humans?"

They gave these AI judges the task of finding the "good, creative parts" and the "bad, nonsensical parts" of a story.

The Good News: The AI judges were much better than random guessing. They could spot some creative gems.
The Bad News: They were terrible at spotting the "nonsensical" parts. They often missed the "tire soup" because, to an AI, weirdness sometimes looks like creativity.

4. The New Way Forward

The paper concludes that we need to stop using "n-gram novelty" as the sole score for creativity. It's like grading a painting only on how many colors the artist used, ignoring whether the picture actually looks like anything.

The Takeaway:
True creativity isn't just about being different; it's about being different and meaningful.

Old Metric: "Is this phrase new?" (AI: "Yes! But it's nonsense.")
New Metric: "Is this phrase new AND does it make the story better?" (AI: "Sometimes yes, but often no.")

The authors suggest that instead of just counting how "rare" a phrase is, we need to use smarter AI judges that understand context and meaning, just like a human editor does. They proved that while AI can write, it still struggles to write well without sounding like a broken robot trying too hard to be unique.

1. Problem Statement

The paper addresses a critical flaw in current methods for evaluating textual creativity in Large Language Models (LLMs). Recently, metrics like the CREATIVITY INDEX have relied heavily on n-gram novelty (measuring how rarely a text fragment appears in pre-training corpora) to quantify creativity.

The authors argue that this approach is theoretically inadequate based on the standard psychological definition of creativity, which requires two components:

Novelty: The text must be original.
Appropriateness: The text must be sensical and pragmatic (fitting the context).

The paper posits that high n-gram novelty does not guarantee creativity; an LLM can generate highly novel text that is nonsensical or contextually incoherent ("AI slop"). Conversely, text can be creative even if it uses common n-grams. The study aims to empirically test the relationship between n-gram novelty and human-judged creativity, and to determine if LLMs can effectively replace human experts in identifying creative or flawed expressions.

2. Methodology

Data Collection and Annotation

Corpus: The authors collected 50 fiction passages from The New Yorker (human-written) and generated 50 corresponding passages using open-source models (OLMo-7B and OLMo-2-32B) and frontier models (GPT-5, Claude-4.1).
Annotation Task: 26 professional writers (MFA graduates and published authors) performed a "close reading" task.
- Unit of Analysis: Passages were split into "atomic" expressions based on punctuation.
- Metrics: Annotators rated expressions on three dimensions:
  1. Sensicality: Does the expression make sense standalone?
  2. Pragmaticality: Does the expression make sense in the specific context?
  3. Perceived Novelty: Is the expression unusual, surprising, or original?
- Creative Definition: An expression is deemed "creative" only if it is simultaneously judged as sensical, pragmatic, and novel.
Dataset Size: 8,618 annotated expressions across 2,783 unique expressions.

Technical Metrics and Modeling

n-gram Novelty: Measured using the Infini-gram package to calculate perplexity ( $P_\infty$ ) against the specific training corpora of the models (Dolma-v1.7 for OLMo, v4 for OLMo-2). Lower perplexity indicates higher familiarity (low novelty); higher perplexity indicates novelty.
Statistical Analysis: The authors employed hierarchical/multilevel logistic regression (mixed-effects models) to account for:
- Random intercepts for individual raters (to handle subjectivity).
- Random intercepts for seed passages (to handle topic variation).
- Fixed effects for generation source (Human vs. AI) and perplexity.
LLM-as-a-Judge (LLM-J) Evaluation: The authors tested whether frontier models (GPT-5, Claude-4.5, Gemini-2.5/3 Pro) and fine-tuned open-source models could replicate human judgments in identifying:
- Perceived Novel Expressions (Positive aspect).
- Non-Pragmatic Expressions (Negative aspect).
- Evaluation was done via Zero-shot, Few-shot, and Fine-tuned (LoRA/API) settings, measured by F1 score against human ground truth.

3. Key Contributions

Empirical Refutation of n-gram Novelty as a Sole Metric: The study provides robust evidence that n-gram novelty is a poor proxy for creativity.
The "Novelty-Pragmaticity" Trade-off in AI: The authors discovered a negative correlation between n-gram novelty and pragmaticality in AI-generated text, a phenomenon not observed in human writing.
Human vs. AI Creativity Gap: Frontier models are significantly less likely to produce expressions judged as creative compared to humans.
LLM-as-a-Judge Viability: While LLMs struggle to identify non-pragmatic (flawed) text, they show strong alignment with expert preferences regarding novelty, outperforming traditional n-gram metrics.

4. Key Results

A. The Failure of n-gram Novelty

Low Precision: Approximately 91% of expressions in the top quartile of n-gram novelty were not judged as creative by human experts.
Low Recall: Conversely, about 25% of unique creative expressions had very low perplexity (low n-gram novelty), proving that creativity often relies on familiar phrasing used in novel contexts.
Conclusion: Relying solely on n-gram novelty leads to a massive false positive rate for creativity.

B. The Negative Impact on Pragmaticity

Open-Source Models: As n-gram novelty increases in OLMo and OLMo-2 outputs, the probability of the expression being rated as pragmatic decreases significantly ( $\beta = -0.17$ for OLMo, $\beta = -0.48$ for OLMo-2).
Human Baseline: No such negative correlation exists for human-written text; humans can be novel without sacrificing context.
Implication: Optimizing LLMs for novelty (e.g., via high temperature or specific decoding strategies) may inadvertently degrade the coherence and utility of the text.

C. Human vs. AI Creativity

Frontier models (GPT-5, Claude-4.1) and open-source models are significantly less likely to generate creative expressions than humans (Odds Ratios < 0.6).
Even with high novelty, AI text is less likely to be deemed "creative" because it often fails the pragmaticity/sensicality check.

D. LLM-as-a-Judge Performance

Novelty Detection: Reasoning models (e.g., GPT-5) achieved an F1 score of ~41.3 for identifying novel expressions, roughly 4x higher than a random baseline.
Non-Pragmatic Detection: Models struggled significantly here, with F1 scores < 20. This aligns with the finding that LLMs themselves tend to produce non-pragmatic text when attempting novelty.
Alignment with Preferences:
- LLM-J novelty scores correlated strongly with expert preferences (outperforming the CREATIVITY INDEX metric).
- LLM-J pragmaticity scores correlated with crowd preferences but failed to align with expert preferences, suggesting experts are more sensitive to subtle pragmatic flaws that LLMs miss.

5. Significance and Implications

Redefining Creativity Metrics: The paper argues for a shift from single-metric novelty scores to multi-dimensional frameworks that explicitly penalize non-pragmatic or nonsensical output.
Warning Against "Novelty Optimization": Developers optimizing LLMs for diversity or novelty must be cautious, as this may degrade the "appropriateness" required for genuine creativity.
Evaluation Pipeline: The study suggests that while LLMs can serve as effective judges for novelty, they are currently insufficient for detecting contextual incoherence (non-pragmaticity) without human oversight.
Dataset Release: The authors release a high-quality dataset of 8,618 expert-annotated expressions, providing a new benchmark for evaluating textual creativity and the "AI slop" phenomenon.

In summary, the paper concludes that n-gram novelty is a necessary but insufficient condition for creativity. True textual creativity requires a balance of originality and contextual appropriateness, a balance that current LLMs struggle to maintain when pushed toward high novelty, and which n-gram metrics fail to capture.