💬 NLP

Literary Non-Style in LLM-Generated Text

This paper argues that the distinct "non-style" of LLM-generated text arises from specific statistical n-gram patterns that reveal semantic limitations, demonstrating that stylistic and semantic qualities in AI writing are fundamentally intertwined rather than separable.

Original authors: Cory Massaro

Published 2026-07-21

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Cory Massaro

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to figure out if a story was written by a human or a robot. You aren't looking for spelling mistakes or grammar errors; you are looking for the "soul" of the writing. In the world of language science, there is a famous rule called Zipf's Law. Think of it like a playlist of your favorite songs. In any big collection of music, a few hits (like "Happy Birthday" or a massive pop song) get played over and over again, while most songs are played only once or twice. If you rank these songs from "most played" to "least played," the drop-off follows a very specific, predictable curve. This happens in human speech, in books, and even in the way we use words. Scientists have long believed that this "musical" pattern is a sign of natural, human creativity. But now, with computers that can write stories, we have to ask: Do these machines follow the same musical rules, or do they sound like a broken record?

This paper, written by Cory Massaro, investigates exactly that. The author wants to know if text generated by Large Language Models (LLMs)—the super-smart AI chatbots we see everywhere—feels different from text written by humans. The study suggests that while AI can mimic the words we use, it fails at the rhythm and variety that make human writing feel alive. By counting how often specific groups of words (like "three-word phrases" or "four-word phrases") appear, the author found that AI writing is much more repetitive and predictable than human writing. It's as if the AI is stuck in a loop, recycling the same favorite phrases, whereas a human writer keeps surprising you with new combinations. The paper concludes that this lack of variety isn't just a style issue; it actually limits the ideas the AI can express. If you can't say things in different ways, you can't say as many different things at all.

The Story of the "Robot Rhythm"

To understand what the author found, let's look at the "ingredients" of the story. The paper compares two main types of writing:

The Human: A classic memoir called The Education of Henry Adams, written by a real person in the early 1900s.
The Robot: A book called The Inner Life of an AI, which was "written" by an AI (ChatGPT) but put together by a human prompter.

The author also looked at a massive dataset containing 8 million words of human writing and 8 million words of AI writing to make sure the results weren't just a fluke.

The Big Discovery: The "Stock Phrase" Trap
The author noticed something strange when reading the AI's book. It felt like the robot was using a set of "canned" phrases over and over again. For example, the phrase "despite these challenges" appeared 16 times in the book, and "I began to" showed up over 50 times! In contrast, the human author, Henry Adams, used a much wider variety of words. He rarely repeated the same four-word sequence.

To prove this wasn't just a feeling, the author did some math. They counted how many unique groups of words (called n-grams) appeared in the texts.

1-grams are single words.
2-grams are two-word pairs (like "very good").
3-grams and 4-grams are longer chains.

The results were clear:

In the human text, the variety was high. For 4-word phrases, the ratio of unique phrases to total phrases was 0.9890. This means Henry Adams almost never repeated a four-word sequence.
In the AI text, the variety was much lower. The ratio for 4-word phrases was only 0.6882. This means the AI repeated its favorite four-word chains constantly.

The "Slope" of the Story
The author also looked at the "slope" of the writing, which is a fancy way of describing how quickly the writing runs out of new ideas. In a perfect human story, you keep finding new words and phrases, but they appear less and less often, following a smooth curve (Zipf's Law).

For the AI, the curve was too steep. The most common phrases were too common, and the rare phrases were too rare.
The author found that for 4-word phrases, the AI's "slope" was -0.395 (in one test), while the human's was -0.178. The steeper number for the AI means it clings to its favorite phrases much tighter than a human does.

What About "Stock Phrases"?
The author wondered: "Maybe the AI just uses different words for the same idea? Like saying 'big' instead of 'huge'?" To test this, they "lemmatized" the text, which means grouping words by their root (so "running," "ran," and "runs" all count as the same word).

The Result: Even after grouping similar words together, the AI still sounded repetitive. The "slope" didn't change much. This suggests the problem isn't just about using synonyms; the AI is genuinely stuck in a rut of specific phrases.

The "Broken Record" vs. The "Jazz Solo"

So, what does this all mean? The author uses a great analogy: imagine a human writer is a jazz musician improvising a solo. They might play a few familiar notes, but they constantly mix them up, add new riffs, and surprise the listener. The AI, on the other hand, is like a broken record player that keeps skipping to the same catchy chorus.

The paper argues that this isn't just about the AI sounding "boring." It's about what the AI can say.

The Limit: Because the AI relies so heavily on the same few phrases, it has a smaller "toolbox" of ideas. If you have a fixed number of words to write a story, a human can pack in more different ideas because they use more unique combinations. The AI, stuck in its loop, ends up saying the same things in slightly different ways.
The "Feel": The author notes that people can often "feel" when they are reading AI text. It feels flat, formulaic, and a bit hollow. This study suggests that feeling comes from a lack of phrasal variety. The AI is missing the "spark" of human creativity because it can't break its own patterns.

What the Paper Says It Doesn't Know

It's important to note what this study didn't do. The author didn't claim that AI writing is "bad" in a moral sense, nor did they say AI can never write good literature. They simply measured the statistics of the words.

They didn't prove that AI can never learn to be more creative; they just showed that right now, in the texts they analyzed, the patterns are very different from humans.
They didn't say that repetition is always bad (sometimes repeating a phrase is a powerful artistic choice, like in a poem). But the AI repeats phrases too much and in the wrong places, like a robot trying to be a poet but forgetting how to vary its rhythm.

The Final Takeaway

In the end, the paper suggests that style and meaning are connected. You can't separate the "feel" of a story from the "ideas" in it. If a writer (or a robot) can't find new ways to say things, they are limited in what they can express. The AI's "broken record" rhythm isn't just a stylistic quirk; it's a sign that the machine's ability to express complex, varied human thoughts is still, well, a work in progress. The numbers show that while the AI can talk, it hasn't quite learned how to sing with the same wild, unpredictable variety that humans do.

Technical Summary: Literary Non-Style in LLM-Generated Text

Problem Statement
Prior research has established that Large Language Model (LLM)-generated text exhibits quantitative and qualitative deviations from human writing, characterized by a distinct textual "feel" and a restricted semantic range. While stylistic differences are often noted anecdotally, the relationship between these stylistic deficiencies and semantic limitations remains under-explored. The central problem addressed is whether the statistical distribution of n-grams in LLM text reveals specific structural deficits that correlate with a lack of phrasal variety, thereby constraining the model's expressive range and semantic content. The paper posits that style and semantics are not cleanly separable; an impoverished lexical and formulaic register indirectly limits the ideas that can be expressed.

Methodology
The study employs a comparative statistical analysis of n-gram distributions across four distinct text corpora:

LLM-Generated: The Inner Life of an AI: A Memoir by ChatGPT (31,147 tokens) and a subset of Zachary Grinberg's Human vs. LLM Text Corpus (8,000,005 tokens).
Human-Generated: The Education of Henry Adams (32,984 tokens, selected to match the size of the LLM memoir) and a random subset of the human portion of Grinberg's corpus (8,000,010 tokens).
Control/Artificial: Samuel Beckett's Lessness, used as a benchmark for highly artificial, constrained text.

The analysis focuses on the frequency distribution of 1-grams through 4-grams. The author calculates:

Type/Token Ratios: To measure lexical and phrasal diversity.
Zipfian Slopes: The slope ( $m$ ) of the linear relation $\log(\text{frequency}) = m \log(\text{rank})$ .
Zipf-Mandelbrot Parameters: Fitting the distribution to the formula $\text{frequency} = \frac{k}{(\text{rank} + \alpha)^\beta}$ .

To ensure robustness, these metrics are computed across four variations of the corpora: with/without hapax legomena (single-occurrence types) and with/without lemmatization.

Key Contributions and Hypotheses
The paper tests two primary hypotheses regarding the statistical properties of LLM text:

Hypothesis 1: The Zipfian characteristics of higher-order n-grams in LLM text will differ from human text, specifically exhibiting steeper slopes (higher absolute values) in the log-log plot, indicating a concentration of weight on common phrases.
Hypothesis 2: Because LLMs operate on sub-word embeddings that allow for synonym substitution, the slope differences between LLM and human text should be even more exaggerated when comparing lemmatized versions of the corpora.

Results
The analysis yields the following findings:

Type/Token Ratios: At all n-gram levels (1 through 4), human-generated corpora exhibit higher type/token ratios than LLM-generated corpora. For instance, in the 4-gram analysis, The Education of Henry Adams has a ratio of 0.9890, whereas the LLM corpus has a ratio of 0.6882. This indicates that human text rarely repeats four-word sequences, while LLM text demonstrates significant homogeneity.
Zipfian Slopes: As predicted by Hypothesis 1, LLM-generated corpora consistently exhibit steeper slopes than comparable human corpora. This divergence becomes more pronounced at higher n-gram levels (3-grams and 4-grams). For example, in the +hapax+lemmatized condition, the 4-gram slope for the LLM corpus is -0.395, compared to -0.201 for the human corpus.
Lemmatization Effects: Contrary to Hypothesis 2, lemmatization had only negligible effects on the slope differences. The divergence between LLM and human distributions persisted regardless of whether tokens were lemmatized, suggesting the issue is not merely a failure to recognize synonyms but a fundamental overreliance on specific phrasal sequences.
Qualitative Observations: The LLM text relies heavily on stock transitional phrases (e.g., "as an AI" appears 110 times in the memoir, often at sentence beginnings; "one of the most" appears 2,787 times in the large LLM corpus vs. 870 in the human corpus). In contrast, human text utilizes varied diction, metaphor, and syntax.

Significance and Claims
The paper concludes that LLMs suffer from a "literary non-style" characterized by a lack of phrasal variety. The study demonstrates that:

Style and Semantics are Interconnected: The statistical over-reliance on common n-grams (low phrasal variety) directly correlates with a reduced semantic range. If an LLM cannot compose aesthetically coherent prose with varied phrasing, its ability to express distinct ideas is impaired.
Expressive Limitations: Given a fixed word limit, a human author is likely to express more distinct ideas than an LLM because the human model generates a wider variety of word sequences, whereas the LLM model gravitates toward a narrower set of high-probability sequences.
Distinction from Artificial Constraints: While LLM text trends toward the mechanistic repetition seen in Beckett's Lessness, it does not reach the same extreme levels of artificiality; however, the direction of the trend is clear.

The author asserts that these distributional differences are not merely aesthetic but represent a fundamental constraint on the "feel" and semantic capacity of LLM-generated text, validating the claim that stylistic flatness is a symptom of impoverished semantic expression.

The Story of the "Robot Rhythm"

The "Broken Record" vs. The "Jazz Solo"

What the Paper Says It Doesn't Know

The Final Takeaway

Technical Summary: Literary Non-Style in LLM-Generated Text

More like this