🤖 machine learning

Internal Data Repetition Destroys Language Models

This paper demonstrates that in the Chinchilla-era scaling paradigm, data repetition systematically harms language model performance by creating a compute-equivalent loss peak at an intermediate repeat count that scales with model size, a phenomenon analytically explained as a statistical tradeoff between memorization and generalization.

Original authors: Jessica Chudnovsky, Joshua Kazdan, Noam Levi, Rylan Schaeffer, Yegor Denisov-Blanch, Bo He, Mehmet Donmez, Sanmi Koyejo, David Donoho

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jessica Chudnovsky, Joshua Kazdan, Noam Levi, Rylan Schaeffer, Yegor Denisov-Blanch, Bo He, Mehmet Donmez, Sanmi Koyejo, David Donoho

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a student (a Language Model) to write essays by giving them a massive library of books to read. The goal is to make them as smart as possible using a limited amount of study time (computing power).

This paper investigates what happens when that library isn't perfect and contains duplicate books. Specifically, it asks: Does it matter if we give the student the same book twice, or the same book 100 times?

Here is the breakdown of their findings using simple analogies:

1. The Problem: The "Echo Chamber" Effect

In the past, researchers thought that if you just removed exact duplicates from your training data, you were safe. But in reality, even "clean" data has some repeats.

The researchers found that repetition doesn't just make learning slightly slower; it can actively destroy the model's performance in a very specific, counter-intuitive way. It's not a straight line where "more repeats = worse results." Instead, it's a trap.

2. The "Goldilocks" Trap (The Worst-Case Scenario)

The most surprising finding is that the damage isn't caused by repeating a tiny bit of data a million times, nor by repeating a huge chunk of data just twice.

The Analogy: Imagine a student studying for a test.

Scenario A: They read 1,000 unique books once. (Great results).
Scenario B: They read 1,000 unique books, but they also re-read one specific book 10,000 times. (The student memorizes that one book perfectly but fails to learn general concepts. Bad results).
Scenario C (The Trap): They read 1,000 unique books, but they re-read a medium-sized pile of 50 books about 100 times each.

The paper found that Scenario C is the worst. It's like the student gets stuck in a loop where they memorize that medium pile of books so thoroughly that they stop learning anything new from the rest of the library. They become "over-specialized" on the duplicates and "under-specialized" on the unique data.

Too few repeats: The student ignores the duplicates and learns normally.
Too many repeats: The student memorizes the duplicates so hard they effectively "tune out" the rest of the data, but the sheer volume of unique data still helps.
Just the right amount of repeats: The student gets confused. They spend too much time on the duplicates to memorize them, but not enough time on the unique data to learn general rules. This "middle ground" causes the biggest drop in intelligence.

3. The "Size Matters" Rule

The researchers discovered that the size of this "danger zone" changes depending on how big the student (the AI model) is.

Small models get hurt when they repeat a small pile of books many times.
Large models get hurt when they repeat a much larger pile of books, but fewer times.

Think of it like a sponge. A small sponge (small model) gets clogged if you pour a little bit of mud on it too many times. A giant sponge (large model) can handle a little mud, but if you pour a huge bucket of mud on it a moderate number of times, it gets clogged instantly.

4. The Cost: Wasted Money and Energy

The paper translates this "stupidity" into money and energy. They found that if you hit this "worst-case" repetition trap, you might as well have thrown away 33% of your computing budget.

The Analogy: Imagine you paid for a 100-hour study session. Because of the bad repetition structure, the student only learned as much as they would have in a 67-hour session. You wasted 33 hours of electricity and money for no gain.

5. Why Does This Happen? (The Simple Math)

You might think this is a complex quirk of how AI brains (Transformers) work. But the authors proved it's actually a basic statistical problem.

They built a simple math model (like a basic linear regression) that doesn't even use AI. When they fed this simple math model duplicate data, it showed the exact same "Goldilocks" peak of bad performance.

The Lesson: This isn't a bug in the AI's fancy attention mechanisms; it's a fundamental law of statistics. When you have a mix of unique data and repeated data, there is a specific balance where the model gets confused between "memorizing the copy" and "learning the general rule," and that confusion hurts the most.

Summary

Repetition is bad, but not in the way you think.
The worst damage happens when you repeat a medium-sized chunk of data a moderate number of times.
Bigger models are vulnerable to repeating larger chunks of data.
The cost is huge: You can waste a third of your computing power just by having the wrong repetition structure.
The cause is a basic statistical trade-off between memorizing duplicates and learning general patterns, not a complex AI failure.

The paper concludes that simply removing duplicates isn't enough; we need to understand how the remaining duplicates are structured to avoid wasting massive amounts of resources.

Technical Summary: Internal Data Repetition Destroys Language Models

Problem Statement
As pretraining for language models (LMs) enters a data-constrained regime, high-quality public corpora are exhausted, necessitating multi-epoch training and the use of imperfectly deduplicated datasets. While aggressive deduplication is standard, residual exact duplicates, paraphrased templates, and semantically redundant pages remain. Previous controlled studies on data repetition, such as Hernandez et al. [11], predated the Chinchilla-style compute-optimal scaling laws. Consequently, they measured the cost of repetition indirectly via "effective parameter count" reductions and on fixed token budgets that undertrained large models and overtrained small ones. This paper addresses the need to quantify the cost of exact document-level repetition within the modern Chinchilla paradigm, where performance is measured in FLOPs rather than parameter counts, and training budgets are optimized for compute efficiency.

Methodology
The authors conduct a controlled study isolating the impact of repetition structure while holding the total compute budget and the fraction of repeated tokens constant.

Experimental Setup: The study trains Qwen3-style decoder-only transformers with parameter counts $N \in \{34, 48, 63, 93, 153, 344\}$ M on the FineWeb-Edu-Dedup corpus.
Repetition Protocol: For each run, 10% of the training tokens ( $f=0.1$ ) are drawn from a specific pool of documents, while 90% come from unique documents. The structure of the repetition is varied by changing the repeat count $R$ (how many times each document in the pool is replayed) and the pool size $D_r$ . This creates an iso-FLOP curve where the total compute $C = 6NT$ remains fixed, but the concentration of repeated data changes.
Metrics: The authors introduce Compute-Equivalent Gain (CEG) and Compute-Equivalent Loss (CEL). CEG is defined as the ratio of the compute a no-repetition run would need to achieve the same loss as a repeated-data run, divided by the actual compute spent. $CEL = 1 - CEG$. A value of $CEG < 1$ indicates that repetition caused a loss in compute efficiency.
Baseline: A fitted no-repetition Chinchilla scaling law, $L(C) = E + KC^{-\gamma}$ , is derived from six $OT=1$ (Chinchilla-optimal) baselines to convert evaluation losses into equivalent compute costs.
Theoretical Analysis: To understand the mechanism, the authors derive closed-form expressions for train and test loss in a misspecified linear regression model with verbatim duplicates, analyzing the block-diagonal noise covariance introduced by repeated samples.

Key Results

Non-Monotonic Damage Peak: Evaluation loss is not monotonic with respect to the repeat count $R$ . Instead, damage peaks at an intermediate repeat count. Repeating a moderately sized subset a moderate number of times causes more harm than repeating a very small subset many times or a large subset a few times.
Scaling Law for Peak Location: The location of this peak ( $R_{peak}$ ) follows a power law with respect to model size $N$ and total compute $C$ . Specifically, $R_{peak} \propto N^{-0.96}$ and $R_{peak} \propto C^{-0.25}$ . This implies that as models scale up, the most damaging repetition structure involves fewer repeats of a larger pool of documents.
Significant Compute-Equivalent Loss: At the largest scale tested (344M parameters, $OT=1$), the worst-case repetition structure results in a CEG of approximately 0.67, corresponding to a CEL of 0.33. This means the run achieves the same loss as a no-repetition run trained with only two-thirds of the FLOPs.
Statistical Mechanism: The non-monotonic peak is reproduced in a simple misspecified linear regression model. The authors show that this phenomenon arises from a statistical tradeoff between memorization and generalization in the presence of duplicated samples, suggesting the effect is not specific to transformer architectures, attention mechanisms, or optimizers.

Significance and Claims
The paper claims to modernize the study of data repetition by shifting the metric from "effective parameters" to "compute-equivalent units," aligning with current industry practices.

Precision in Cost Quantification: The findings allow practitioners to quantify the wasted compute incurred by the specific structure of duplicates in pretraining corpora, not just the volume of duplicates.
Predictability: The identified power law ( $R_{peak} \propto N^{-0.96}$ ) provides a heuristic for predicting the most harmful repetition configurations for a given model size, enabling better data curation strategies.
Generalizability: By demonstrating that the same qualitative loss peak occurs in a simple linear regression model, the authors argue that the damage from repetition is a fundamental statistical property of duplicated samples in misspecified learners, rather than an artifact of specific deep learning components.

The authors remain modest regarding the absolute values of CEG and CEL, noting they depend on a three-parameter scaling law fit to a limited number of baselines. However, they assert that the qualitative non-monotonicity and the existence of a worst-case repetition structure are robust findings. The work concludes that simply reporting the fraction of duplicated data is insufficient; the concentration and repetition count of those duplicates are critical determinants of training efficiency.