🤖 AI

Scaling Domain Data Repetition in LLM Pretraining

This paper investigates the trade-off between data repetition and overfitting in large language model pretraining, revealing that optimal repetition counts for high-quality domain data mildly increase with model size at a fixed tokens-per-parameter ratio and are strongly negatively correlated with a domain's validation loss, suggesting that tuning on smaller proxy models can effectively guide scaling strategies for larger models.

Original authors: Jingwei Li, Xinran Gu, Rui Dai, Xintong Hao, Chengyin Xu, Yan Wu, Shuran Zheng, Jingzhao Zhang

Published 2026-08-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jingwei Li, Xinran Gu, Rui Dai, Xintong Hao, Chengyin Xu, Yan Wu, Shuran Zheng, Jingzhao Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to speak, write, and think. To do this, you feed it a massive library of text, a process called "pretraining." But here's the catch: the robot gets bigger and smarter as you add more "brain cells" (parameters), and to keep it from getting confused or underperforming, you have to feed it even more words. This is the world of Large Language Models (LLMs). The problem is that while you can easily find endless amounts of generic internet text (like random forum posts or news articles), finding truly high-quality, specialized knowledge (like advanced math, coding, or medical facts) is like finding a needle in a haystack. There just isn't enough of it to feed a giant robot.

So, what do you do when you run out of the good stuff? You start repeating it. You take that precious math textbook and feed it to the robot again and again. But this is a delicate balancing act. If you repeat the good stuff too much, the robot might just memorize the book word-for-word and fail to understand the actual concepts, a problem called "overfitting." If you don't repeat it enough, the robot gets drowned out by the sea of generic internet noise and never learns the hard stuff. The big question is: How many times should you repeat the good data before the robot starts to get confused?

This paper, titled "Scaling Domain Data Repetition in LLM Pretraining," dives deep into that exact question. The researchers from Tsinghua University and ByteDance Seed wanted to figure out the perfect recipe for mixing high-quality data with general data as robots get bigger. They tested this on four special "flavors" of data: Code, Math, Wikipedia, and Medical records.

Here is what they discovered, and it's a bit of a plot twist. For a long time, people thought that as robots got bigger, they became more fragile and would overfit (memorize) data faster. The old advice was: "Don't repeat the good stuff too much, or the big robot will break." But the authors found something surprising when they kept the training budget proportional to the robot's size (a rule called a fixed "tokens-per-parameter" ratio). They found that bigger robots can actually handle more repetition than smaller ones. It's like a giant athlete who can run more laps without getting tired compared to a smaller one, provided they are both training with the same intensity.

However, the most important rule they found isn't about the size of the robot, but about the quality of the data itself. They discovered a strong link between how well a robot learns a specific topic and how much it can repeat that topic. If the robot learns a domain (like Math) very well and gets a low "error score" (validation loss), it can handle repeating that data many times without breaking. But if the robot struggles with a topic and has a high error score, it will overfit very quickly, so you should repeat that data very few times.

Interestingly, the amount of unique data you start with doesn't seem to matter much for deciding how many times to repeat it. Whether you have a tiny pile of unique math problems or a huge one, the "sweet spot" for repetition stays roughly the same. The researchers suggest that if you want to train a massive robot, you don't need to guess. You can train a smaller "proxy" robot first, see how well it learns the math or code, and use that result to safely guess how many times to repeat the data for the giant robot.

They also tested what happens if you swap unique data for repeated data while keeping the total amount of "good stuff" the same. They found that for some topics, like Math, repeating the data is almost as good as having more unique data. But for others, like Wikipedia, repeating the data is a bad idea; the robot learns much worse if it just hears the same Wikipedia sentences over and over instead of reading new ones.

Finally, they looked at how the robot's "learning speed" (learning rate) changes during training. They found that if you slow the robot down early in training, it gets more sensitive to repetition and overfits sooner. But if you keep the learning speed high for longer, the robot can tolerate more repetition before it starts to memorize instead of learning.

In short, this paper suggests that there is no single magic number for how many times to repeat data. Instead, the best strategy depends on how well the robot is already learning that specific subject. If it's a fast learner, repeat the data more. If it's a slow learner, stop repeating it sooner. And surprisingly, bigger robots are actually more robust to repetition than we thought, as long as we feed them the right amount of data for their size.

Technical Summary: Scaling Domain Data Repetition in LLM Pretraining

Problem Statement

As Large Language Models (LLMs) scale, maintaining an appropriate tokens-per-parameter ratio (TPP) requires increasing the total training-token budget. While general web data can be scaled relatively easily, high-quality domain-specific data (e.g., code, mathematics, medical texts) is scarce and difficult to scale at the same rate. Consequently, as model size and the total token budget grow, the mixing ratio of high-quality domain data tends to decrease, potentially leading to a failure in acquiring target knowledge if the ratio falls below a critical threshold.

To counteract this dilution, practitioners often repeat available high-quality data. However, this introduces a trade-off: repetition helps maintain the necessary data fraction for knowledge acquisition but increases the risk of overfitting. Existing literature suggests that larger models are more susceptible to overfitting from repeated data, implying that repetition should be minimized. However, these conclusions are often drawn under experimental setups where the total training-data size is fixed across model scales, rather than preserving the TPP. This paper investigates the optimal repetition count under a more practical scaling regime where the training-token budget grows proportionally with model size (fixed TPP).

Methodology

The authors conducted a systematic empirical study across four high-quality domains: Code, Mathematics, Wikipedia, and Medical data.

Scaling Regime: The study adheres to a fixed TPP ( $D/N$ ), where $D$ is the total training tokens and $N$ is the model size. The token budget $D_N$ scales linearly with model size ( $D_N = \text{TPP} \cdot N$ ).
Experimental Setup: For each model size and domain, the authors varied two parameters:
1. Unique Data Fraction ( $\alpha$ ): The proportion of unique high-quality tokens in the total budget (tested at 1/40, 1/20, 1/10).
2. Repetition Count ( $e$ ): The number of times a fixed subset of unique high-quality tokens is revisited (tested from 1 to 7).
Evaluation: Models were evaluated using in-distribution validation loss (on held-out data from the same domain) and out-of-distribution (OOD) pretraining validation loss (on general web data like ArXiv and News).
Theoretical Analysis: To explain empirical findings, the authors employed a one-hot linear regression model in $\mathbb{R}^\infty$ with power-law distributed knowledge units. They analyzed the decomposition of risk into "knowledge-acquisition error" and "noise-fitting error" to derive stopping times (optimal repetition counts) under different scaling conditions.

Key Findings

1. Correlation with Validation Loss

The optimal repetition count is strongly negatively correlated with the final validation loss of a domain (Pearson correlation of -0.944).

Domains that achieve lower validation losses (indicating the model learns the domain well) can tolerate and benefit from more repetitions.
Domains with higher validation losses tend to overfit earlier, requiring fewer repetitions.
This suggests that validation loss serves as a practical proxy for determining safe repetition limits.

2. Impact of Model Size under Fixed TPP

Contrary to findings in fixed-data-budget settings (where optimal repetition decreases as models grow), this study finds that under a fixed TPP, the optimal repetition count mildly increases with model size.

Larger models trained with proportionally larger token budgets benefit from slightly more repetitions before overfitting occurs.
This supports a conservative transfer strategy: a repetition count that does not cause overfitting in a smaller proxy model (trained at the same TPP) is unlikely to cause overfitting in a larger target model.

3. Insensitivity to Unique Data Fraction

The optimal repetition count is largely insensitive to the fraction of unique high-quality data ( $\alpha$ ) within the tested range.

While increasing $\alpha$ consistently lowers the absolute validation loss, the location of the optimal repetition count (the $e$ that minimizes loss) remains nearly unchanged.
This implies that when tuning repetition on a proxy model, the specific fraction of unique data used does not need to match the target deployment exactly; any representative fraction within the tested range suffices.

4. Domain Dependence and OOD Effects

Domain Specificity: The effectiveness of repetition is highly domain-dependent. For example, Mathematics data is robust to repetition (up to 4-5 times), whereas Wikipedia data degrades sharply after 2 repetitions.
OOD Performance: When the total fraction of high-quality tokens is fixed, replacing unique tokens with repeated tokens has a limited effect on out-of-distribution (OOD) performance. The degradation is primarily confined to the in-domain validation loss.

5. Learning Rate Schedule Influence

The optimal repetition count depends on the learning rate schedule. Schedules with earlier decay (e.g., starting decay at 0% of training) lead to degradation after fewer repetitions. Conversely, delaying decay or using a constant learning rate allows the model to tolerate more repetitions, likely due to stronger optimization noise acting as implicit regularization against memorization.

Theoretical Explanation

The authors provide a theoretical framework explaining these observations through the trade-off between knowledge acquisition and noise fitting:

Noise Fitting: Repeated optimization reduces knowledge-acquisition error but increases noise-fitting error. The optimal stopping point is where the marginal benefit of learning signal is overtaken by the cost of fitting noise.
Validation Loss: A lower validation loss implies a lower noise level ( $\sigma^2$ ) relative to the signal. Theoretically, lower noise delays the onset of overfitting, allowing for more repetitions (Theorem 4.2).
Scaling: Under fixed TPP, the data budget grows with model size. This allows the reduction in knowledge-acquisition error to dominate for a longer duration as the model scales, justifying the increase in optimal repetition counts (Theorem 4.4).

Significance and Practical Implications

The paper concludes that repetition counts tuned on smaller proxy models, provided they share the same TPP as the target model, can serve as a practical and conservative estimate for larger models.

The proposed workflow for configuring high-quality domain data in large-scale pretraining is:

Train a smaller proxy model with the same TPP.
Sweep repetition counts for the target domain using any representative fraction of unique data.
Identify the repetition count that minimizes validation loss (or stops just before degradation).
Apply this count to the larger target model, as the trend suggests larger models will not overfit earlier than the proxy under the same scaling regime.

This approach offers a data-efficient strategy to maintain high-quality domain representation in LLM pretraining without requiring massive, unique datasets that are often unavailable.