💬 NLP

Prescriptive Scaling Laws for Data Constrained Training

This paper proposes a new prescriptive scaling law that accounts for data repetition and overfitting in data-constrained regimes, demonstrating that beyond a certain point, increasing model capacity is more effective than further data repetition and that strong weight decay significantly mitigates overfitting.

Original authors: Justin Lovelace, Christian Belardi, Srivatsa Kundurthy, Shriya Sudhakar, Kilian Q. Weinberger

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Justin Lovelace, Christian Belardi, Srivatsa Kundurthy, Shriya Sudhakar, Kilian Q. Weinberger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a student for a big exam. You have a limited library of textbooks (high-quality data), but you have an unlimited supply of time and energy (compute).

For a long time, the rule of thumb in AI was: "Just keep reading new pages." The famous Chinchilla law assumed that every single sentence you read was brand new and unique. It told you exactly how many books to read versus how big your brain should be to get the best grade.

But in the real world, good textbooks are rare. Often, you run out of new pages and have to start reading the same books over and over again. The old rules didn't know how to handle this. They thought reading the same page twice was just as good as reading a new one, or at least that it would keep getting slightly better forever.

The Problem: The "Over-Reading" Trap
The authors of this paper discovered that reading the same data too many times is like a student who memorizes the exact font and typos of a textbook instead of understanding the concepts. This is called overfitting.

If you force a student to read the same 100 pages 16 times, they stop learning new things and start getting confused or rigid. The bigger the student (the AI model), the faster they get stuck in this trap. The old rules failed to predict that at a certain point, reading more repetitions actually makes the student worse.

The Solution: A New Rulebook
The authors created a new, simple rule (a "scaling law") that adds a "penalty" for repetition. Think of it like a teacher's note in the margin: "Every time you re-read a page, you lose a little bit of your ability to learn new things."

Their new formula has three parts:

The Floor: Some things are just hard to learn (like guessing the next word in a sentence).
The Brain Size: If your brain is too small, you can't hold all the patterns.
The Repetition Penalty: If you read the same data too many times, you get "stale."

The Big Discovery: Stop Reading, Start Thinking
The most surprising finding is about how to spend your time (compute).

Old Advice: "If you run out of books, just keep reading the same ones over and over."
New Advice: "Once you've read the books a few times, stop."

The paper shows that if you have a fixed library of books, there is a "sweet spot" for how many times to read them. If you have a lot of time (compute) but a small library, the best move isn't to keep reading the same books 20 times. Instead, you should build a bigger brain (a larger model) and read the books fewer times.

It's like realizing that spending 10 hours re-reading a single chapter is a waste of time. It's better to spend that time building a bigger, smarter brain that can understand the chapter in just one or two reads.

The Secret Weapon: Stronger "Discipline"
The paper also tested a technique called weight decay (a way to keep the model from getting too confident or rigid). They found that using a very strong version of this "discipline" is like giving the student a better study method.

It reduces the "stale" penalty by about 70%.
This means the student can handle reading the same books more times without getting confused.
It explains why, in data-scarce situations, using very strong discipline (which was previously thought to be too harsh) actually works better than standard methods.

In Summary
This paper tells us that in the world of AI, data is the bottleneck, not computer power.

Don't over-rehearse: Repeating data too much hurts performance, especially for big models.
Go bigger, not longer: When you run out of new data, spend your computing power on making the model bigger, not on reading the same data more times.
Use strong discipline: Turning up the "weight decay" helps models resist the confusion of re-reading data, allowing them to learn more effectively from limited resources.

The authors proved this by training over 300 different models and showing that their new rulebook predicts the best results much better than the old ones.

Technical Summary: Prescriptive Scaling Laws for Data Constrained Training

Problem Statement

The training of large language models (LLMs) is increasingly constrained by the availability of high-quality data rather than compute. While raw text is abundant, state-of-the-art pipelines rely on aggressive quality filtering, curated subsets, and domain-specific corpora, creating a bottleneck where the compute budget often exceeds the available unique data. In this regime, practitioners frequently resort to training models for multiple epochs over the same data.

Existing scaling laws, particularly the widely adopted Chinchilla law (Hoffmann et al., 2022), assume every training token is unique. Extensions to handle data repetition, such as the effective-data formulation by Muennighoff et al. (2023), model diminishing returns but fail to capture the regime where loss increases due to overfitting. Crucially, these existing models do not explicitly account for the interaction between model size and repetition: larger models overfit faster on repeated data than smaller ones. Without modeling this interaction, scaling laws cannot accurately predict behavior or guide compute-optimal allocation in data-constrained settings.

Methodology

The authors propose a new scaling law formulation that explicitly models the cost of data repetition through an additive overfitting penalty.

Experimental Setup

Models: Over 300 decoder-only language models based on the Llama 2 architecture, ranging from 15M to 1B parameters.
Data: Trained on the FineWeb dataset (Penedo et al., 2024).
Grid: The study sweeps across unique data budgets ( $U_D$ ) from 50M to 6B tokens and repetition counts ( $R_D$ ) from 0 to 15 epochs.
Regularization: Two weight decay strengths were tested: standard ( $\lambda = 0.1$ ) and strong ( $\lambda = 1.0$ ), with all other hyperparameters held constant.
Evaluation: Final validation loss, perplexity, and downstream performance via the OLMES evaluation suite (Gu et al., 2025) across 19 tasks.

Proposed Scaling Law

The authors derive a family of scaling laws that augment the Chinchilla form with an additive penalty term. The general form is:
$L(N, U_D, R_D) = E + \frac{A}{N^\alpha} + \frac{B}{(U_D \cdot (1 + R_D))^\beta} + \text{Penalty}(N, U_D, R_D)$

Where the penalty term captures the excess loss due to repetition. The authors present a "complexity ladder" of forms:

1-Parameter Form: A linear penalty in repetition count scaled by the ratio of model size to unique data: $P \cdot R_D \cdot \frac{N}{U_D}$ .
2-Parameter Form: Introduces a non-linear exponent $\kappa$ on the capacity ratio: $P \cdot R_D \cdot (\frac{N}{U_D})^\kappa$ .
4-Parameter Form: Adds a superlinear exponent $\delta$ on the repetition count and decouples the data-budget exponent $\gamma$ : $P \cdot R_D^\delta \cdot (\frac{N}{U_D^\gamma})^\kappa$ .

This formulation posits that repeated tokens serve a dual role: they continue to reduce the data-sufficiency term (acting as useful data) while simultaneously incurring a growing overfitting cost that depends on model size and data scarcity.

Key Results

1. Superior Fit Quality

The proposed additive penalty laws significantly outperform existing formulations (Chinchilla baseline and Muennighoff et al., 2023) in predicting validation loss under repetition.

Descriptive Fit: Even the simplest 1-parameter additive form substantially outperforms the exponential decay baselines ( $R^2_{multi} \approx 0.95$ vs. $0.58$ for the baseline in the standard study). The 4-parameter form achieves near-perfect fit ( $R^2 \approx 0.997$ ).
Generalization: The additive penalty forms generalize to the held-out Muennighoff et al. (2023) C4 dataset, outperforming their own effective-data formulations on that data, confirming the improvement is not an artifact of the specific training setup.

2. Compute-Optimal Allocation

The new scaling law yields qualitatively different advice for allocating compute in data-constrained regimes compared to prior work:

Prior Work: The Chinchilla law (ignoring overfitting) always recommends more repetition as compute increases. The Muennighoff et al. (2023) law recommends diminishing returns but never suggests stopping repetition.
This Work: The proposed law identifies a compute threshold beyond which further repetition is counterproductive. Beyond this point, the optimal strategy is to allocate additional compute to model capacity (larger $N$ ) while reducing the number of epochs ( $R_D$ ). This creates a "turning back" of the allocation frontier.

3. Prescriptive Validation

The authors validated the law's recommendations by training the specific configurations (model size and epoch count) recommended by each law for fixed compute and data budgets.

The configurations recommended by the proposed law consistently achieved the lowest perplexity and best downstream performance (OLMES BPB) across all tested budgets.
For example, at a compute budget of $2 \times 10^{19}$ FLOPs and $500M$ unique tokens, the proposed law recommended a 2.2B parameter model trained for 3 epochs, outperforming the Chinchilla recommendation (670M params, 10 epochs) and the Muennighoff recommendation (950M params, 7 epochs).

4. Case Study: Weight Decay

The single coefficient $P$ in the penalty term quantifies a configuration's robustness to data repetition.

Reduction in Overfitting: Strong weight decay ( $\lambda = 1.0$ ) reduces the overfitting coefficient $P$ by approximately 70% compared to standard weight decay ( $\lambda = 0.1$ ).
Crossover Point: While strong weight decay incurs a "loss premium" in the single-epoch regime, the reduced overfitting penalty creates a crossover point at high compute budgets ( $C \approx 3.2 \times 10^{18}$ FLOPs for $U_D=250M$ ). Beyond this point, strong weight decay enables the productive use of additional compute, outperforming standard settings. This provides a scaling-law explanation for recent empirical findings (Kim et al., 2026) that optimal weight decay in data-constrained regimes is an order of magnitude larger than standard practice.

Significance and Claims

The paper claims to provide a prescriptive framework for data-constrained training that moves beyond the "infinite data" assumptions of Chinchilla.

New Allocation Advice: The primary contribution is the identification of a regime where repetition becomes detrimental, shifting the optimal strategy from "more epochs" to "larger models."
Quantifiable Robustness: By isolating overfitting into a single coefficient, the law enables direct comparison of training configurations (e.g., different regularization strengths) without exhaustive hyperparameter sweeps.
Practical Guidance: The authors argue that for practitioners with fixed data budgets, there is a specific compute level beyond which repeating data is counterproductive. They provide a concrete method to determine this threshold and the corresponding optimal model size.
Modesty: The authors acknowledge limitations, noting that the fitted exponents are validated up to 1B parameters and 16 epochs, and the law does not encapsulate phenomena like double descent. They also note that the reanalysis of Muennighoff et al. (2023) suggests their disagreement on allocation strategies stems from a mis-specified baseline in the prior work rather than a fundamental contradiction in the physics of scaling.