💬 NLP

Repeated Sequences Reveal Gaps between Large Language Models and Natural Language

This paper introduces a novel evaluation framework based on repeated subsequences and Rényi entropies to demonstrate that, unlike natural language which exhibits stable long-range structural organization, state-of-the-art large language models display systematic and size-dependent deviations in their entropy growth patterns, revealing fundamental gaps in how they capture the deep statistical structure of text beyond surface fluency.

Original authors: Kumiko Tanaka-Ishii

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kumiko Tanaka-Ishii

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Fluency vs. Structure

Imagine you are listening to two people tell a story.

Person A speaks perfectly. Every sentence flows smoothly, the grammar is correct, and they never stutter.
Person B also speaks perfectly, but they have a secret: they are actually just repeating the same few phrases over and over, or they are building a story by stitching together tiny, pre-made Lego bricks without ever creating a new shape.

For a long time, we thought Large Language Models (LLMs) like GPT were getting so good that they sounded exactly like humans. But this paper asks a deeper question: Do they actually understand how to build a long story from the inside out, or are they just really good at sounding smooth in the short term?

The author, Kumiko Tanaka-Ishii, developed a new way to "listen" to these stories not by checking if they make sense grammatically, but by looking at how they reuse their own words and patterns over long distances.

The Tool: The "Echo" Detector

To find the difference, the author uses a concept called Repeated Subsequences.

Think of a text as a long hallway.

Natural Language (Humans): When a human writes a novel, they plant seeds early on. They introduce a character, a theme, or a specific phrase. As the story goes on, they come back to those seeds, water them, and grow new branches. The story "echoes" itself. It reuses old structures to build new meaning.
LLMs (AI): The author found that AI models often act like a broken record or a hall of mirrors. They might repeat things, but they don't seem to have that same deep, structural "echo" where old parts of the story are actively reorganized to support the new parts.

The paper measures this by counting how many times small chunks of text (like 5-letter or 10-letter sequences) appear again later in the document.

The Two Types of Growth: The Garden vs. The Factory

The paper compares how "information" grows as a text gets longer. It identifies two main ways this can happen:

The Power-Law Garden (Natural Language):
Imagine a garden where you keep planting new, unique flowers, but you also keep reusing the same trellis and fence to support them. As the garden grows, the variety of flowers increases, but the structure (the fences and trellises) is reused over and over.
The paper finds that human writing behaves like this. It follows a "Log-Power" pattern. This means the text is constantly recycling its own structural DNA. It's efficient and deeply interconnected.
The Factory Line (AI Models):
Imagine a factory that keeps churning out products. As the factory runs longer, it keeps making slightly different items, but it doesn't seem to have that same deep, recursive reuse of its own internal blueprint.
The paper found that AI models, especially older or smaller ones, behave differently. They show a "Power-Law" pattern that suggests they are constantly generating new structural complexity without the same level of deep, efficient reuse found in human writing.

The Experiment: The "Shuffle" Test

The author tested this on:

Human-written novels (from Project Gutenberg).
AI-generated stories (created by GPT-3.5, GPT-4, and newer models).

They matched the lengths so the comparison was fair. They then ran a mathematical "echo test" (using something called Rényi Entropy) to see how the text reused itself.

The Results:

Humans: No matter how long the book was, the "echo" pattern stayed stable. Humans consistently reused their own structural patterns in a predictable, efficient way.
AI: The AI's pattern changed depending on how "big" the model was.
- Smaller AI models were very chaotic in how they reused (or failed to reuse) structure.
- As the AI models got bigger (like GPT-5), they started to look more like humans, but they still showed a systematic difference. They were getting better at mimicking the "echo," but the underlying math of how they built their stories was still distinct from a human author.

The "Maximal Repetition" Trap

The paper also looked at the longest repeated phrase in a text (like finding the longest sentence that appears twice).

Previous studies focused on these "extreme" repetitions.
The author argues this is like trying to understand a forest by only looking at the tallest tree. It's unstable and misleading.
Instead, looking at all the repeated chunks (the "distribution") gives a much clearer, more stable picture of the text's true structure.

The Bottom Line

This paper doesn't say AI is "bad" or that it can't write good stories. It says that AI and humans build stories differently.

Humans write by constantly referencing and recombining their own past words and structures, creating a tight, efficient web of meaning that grows in a specific mathematical way (Log-Power).
AI writes by predicting the next word, which creates a different kind of growth pattern (closer to Power-Law). Even the smartest AI models haven't quite mastered the human art of "structural recycling" over long distances.

The author concludes that we need new tools to measure AI. We can't just ask, "Did it pass the test?" or "Is it fluent?" We need to ask, "Does it build its world the way a human does?" And right now, the answer is: Not quite.

Technical Summary: Repeated Sequences Reveal Gaps between Large Language Models and Natural Language

Problem Statement
While Large Language Models (LLMs) demonstrate high fluency and coherence in local contexts, it remains an open challenge to determine whether they capture the long-range statistical organization of natural language. Existing evaluation methods, which rely heavily on task-based benchmarks or short-context analyses, provide only indirect evidence regarding the global structure of generated text. Prior work has identified issues such as excessive repetition and reduced diversity in LLM outputs, suggesting that high task performance does not guarantee human-like long-range structure. Specifically, there is a lack of quantitative diagnostics that can distinguish how natural language and LLMs reuse established structural elements over long spans.

Methodology
The authors propose a complementary evaluation framework based on the distribution of repeated subsequences across scales, linking these statistics to higher-order Rényi entropies.

Repeated Subsequence Counting: The method defines the number of repeated blocks ( $D_m$ ) of length $m$ within a sequence of length $n$ . Unlike analyses focusing on maximal repetitions (which can be numerically unstable), this approach examines the distribution of all repeated subsequences.
Higher-Order Rényi Entropy: The authors relate the spectrum of repeated blocks to Rényi entropies ( $H_\alpha$ ) of order $\alpha \ge 2$ . This allows for a finite-length characterization of entropy growth. The empirical entropy $\hat{H}_\alpha(m)$ is calculated based on the frequency of observed block types.
Functional Characterization: The paper models the growth of the effective number of distinguishable blocks ( $\log_2 S_m$ $lo g_{2} S_{m}$ ) using two sublinear forms:
- Power-law ansatz: $\log_2 S_m \propto m^\beta$ , associated with expanding structural degrees of freedom.
- Logarithmic–power ansatz: $\log_2 S_m \propto (\log_2 m)^\gamma$ , associated with strong structural reuse where description length grows slowly relative to sequence length.
Estimation Procedure: A two-stage estimation is employed to handle finite-size effects. First, a correction term ( $\Delta_\alpha$ ) derived from Touchard polynomials is estimated based on the ratio of total blocks to distinct blocks. Second, the exponents $\beta$ and $\gamma$ are fitted to the corrected empirical entropy values.

Experimental Setup
The study compares natural language texts against GPT-generated texts across four model generations: gpt-3.5turbo, gpt-4o-mini, gpt-5-mini, and gpt-5.

Datasets: Natural language texts were sampled from Project Gutenberg (novels) to match the length distributions of the GPT outputs. All texts were preprocessed to remove metadata and non-alphabetic characters.
Controls: Experiments were conducted on 100 texts per category, with length-matched distributions to control for sequence length effects. Additional tests on essays and scientific writing confirmed that trends observed in narratives were consistent across genres.

Key Results

Model Fit: Across datasets, the logarithmic–power model generally provided a better fit ( $R^2 > 0.97$ ) for natural language than the power-law model, particularly for longer texts. This suggests that natural language entropy growth is dominated by structural reuse rather than a continuous expansion of new structural degrees of freedom.
Stability in Natural Language: Natural language exhibited stable entropy-growth patterns (mean exponents $\beta$ and $\gamma$ ) across different datasets and text lengths, despite variability among individual texts. This indicates a form of "weak universality" at the population level.
Systematic Shifts in LLMs: In contrast, GPT-generated texts showed systematic, statistically significant shifts in estimated exponents as model size increased.
- Smaller models (e.g., gpt-3.5turbo) showed a strong preference for the log-power model but with different exponent characteristics.
- As model size increased to gpt-5, both $\beta$ and $\gamma$ increased monotonically, moving away from the stable patterns of natural language.
- Statistical tests (Welch two-sample t-tests) confirmed clear separation between natural language and GPT-5 outputs ( $p \approx 0$ ).
Comparison with Maximal Repetition: While the growth of maximally repeated subsequences (exponent $\eta$ ) showed some convergence between gpt-5 and natural language, the proposed distribution-based method revealed more pronounced and systematic differences in long-range organization.

Key Contributions

Distributional Formulation: The paper proposes a distributional formulation of repetition that enables a finite-length characterization of entropy growth via higher-order Rényi entropies, moving beyond the instability of maximal repetition statistics.
Structural Diagnostic: It demonstrates systematic differences between natural language and LLM-generated text in entropy growth behavior. Specifically, it reveals that while natural language exhibits stable, reuse-dominated patterns, LLMs exhibit systematic shifts in structural organization that correlate with model size.

Significance and Claims
The paper claims that repeated-subsequence entropy serves as a quantitative structural diagnostic capable of revealing systematic differences in long-range organization that are invisible to surface-level fluency or task-based evaluations.

The authors interpret the logarithmic–power scaling in natural language as evidence of "strong structural reuse," where texts are generated through the recombination and re-indexing of shared linguistic resources. In contrast, the shifting exponents in LLMs suggest a weaker manifestation of this "structure-as-reference" organization, potentially due to the next-token prediction objective and finite context utilization. The work establishes that current state-of-the-art LLMs, despite their fluency, do not fully replicate the long-range statistical organization of natural language, providing a new metric for assessing the structural fidelity of generated text.