💻 computer science

The Limits of Training Data Size in Foundation Models: An Empirical Analysis of Quality Filtering under a Fixed Token Pool

Through empirical experiments on a fixed 24-million-token pool, this paper demonstrates that aggressive quality filtering under a fixed compute budget often harms model performance by forcing excessive data repetition, whereas retaining lower-quality data and oversampling high-quality subsets yields superior or equivalent results across various targets.

Original authors: Alexander Memming

Published 2026-07-28

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Alexander Memming

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to speak human language. To do this, you need to feed it a massive library of text—books, websites, articles, and more. For a long time, scientists thought the only thing that mattered was how much text you could feed the robot. The bigger the library, the smarter the robot would get. This idea is called "scaling," and it's the reason our AI assistants are so good today.

But there's a catch. Not all text is created equal. Some pages are full of nonsense, spam, or gibberish, while others are written by experts and are packed with useful information. So, researchers started asking: "Should we throw away the bad stuff and only feed the robot the best books?" This is called "quality filtering." However, there's a second rule in the game: the robot learns best when it sees fresh information. If you make it read the same few pages over and over again, it stops learning and starts just memorizing. This paper explores a tricky balance: if you throw away the bad text, you have fewer pages to read, which means the robot has to read the good pages more times. Does having fewer, better pages help, or does the robot get bored and confused by reading the same thing too many times?

The Great Library Experiment

In this study, a researcher named Alexander Memming set up a clever experiment to solve this puzzle. He didn't use the entire internet (which is too huge to test easily). Instead, he built a "fixed pool" of about 24 million words—a tiny, manageable slice of the web. He then gave this same pool of words to several small AI models, but he changed the rules for each one.

For some models, he kept all the words. For others, he acted like a strict librarian and threw away the bottom half, quarter, eighth, or even just the top sixteenth of the words, keeping only the ones the computer thought were "high quality." He also tried a third approach: instead of throwing anything away, he kept the whole library but told the robot to read the "best" pages many more times than the "okay" pages.

He then tested these robots on three different things:

The Raw Web: A mix of everything, just like the library they were trained on.
The "Edu" Text: High-quality, educational pages (the kind the robot was trained to love).
WikiText: A completely different set of encyclopedia articles the robot had never seen before.

The Surprising Findings

The results were a bit like a rollercoaster, and they changed the rules of the game in three big ways.

1. It Depends on Who You Are Talking To
The biggest surprise was that "quality filtering" doesn't always make the robot smarter. It depends entirely on what you want the robot to do.

If you want the robot to understand the messy, real internet: Throwing away the "bad" text actually hurt the robot. The more the researcher filtered, the worse the robot performed. It's like trying to learn how to drive in a city by only practicing on a perfect, empty racetrack. When you finally hit the real streets with potholes and traffic, you crash. The robot trained on filtered data couldn't handle the messy reality of the web.
If you want the robot to be an expert on high-quality topics: Filtering helped a little bit. If the goal was to sound like a scholar or an encyclopedia, keeping only the best pages made the robot slightly better.

2. The More You Train, The Less You Should Filter
Here is the most important lesson: The size of your budget changes the best strategy.
Imagine you have a small allowance (a small training budget). You might be tempted to buy only the most expensive, high-quality toys. But if you have a huge allowance (a massive budget), buying only the expensive toys means you run out of new toys very quickly. You end up playing with the same expensive toy over and over until you get bored.
The study found that as the training budget grew, the "best" strategy shifted. At first, keeping only the top 1/16th of the data looked okay. But as the robot was trained longer (reading the data 8.2 times instead of 4.1 times), that tiny, filtered library became a disaster. The robot got bored and started making mistakes. The best strategy for a big budget turned out to be keeping more of the data, even the messy parts, so the robot could keep seeing fresh things.

3. Don't Throw It Away, Just Read It More
The researcher also compared "throwing away" the bad text versus "reading the good text more often."

The "Throw Away" method: You delete the low-quality pages.
The "Read More" method: You keep all the pages, but you tell the robot, "Hey, read these top pages 16 times, but only read the bottom pages once."

When the researcher tried to be very aggressive (keeping only the top 1/16th), the "Read More" method won by a huge margin. It was up to 0.39 nats (a unit of measurement for how well the robot predicts the next word) better than the "Throw Away" method.
Think of it this way: If you have a favorite song, you can either delete all the other songs in your playlist and listen to that one song 16 times (Throw Away), or you can keep your whole playlist but make sure that favorite song plays 16 times more often (Read More). The second way is better because you still have the other songs in the background to keep things interesting, and you don't lose the unique variety of the full library.

The Bottom Line

This paper doesn't say we should stop trying to make AI smarter. Instead, it warns us that we can't just blindly throw away "bad" data.

Don't filter if you want to understand the real world: If you want an AI that understands the messy internet, keep the messy data.
Don't filter too much if you have a big budget: If you have a lot of computing power, you need a huge variety of data. If you filter too strictly, you force the AI to repeat itself, and that stops it from learning.
Repetition is better than deletion: If you really want to focus on the best data, don't delete the rest. Just make sure the AI sees the good stuff more often than the bad stuff.

The study was done on small models and a small slice of data, so we can't say for sure exactly what will happen with the giant AI models of the future. But the lesson is clear: treating data quality and data quantity as separate things is a mistake. They are linked, and the best way to train an AI changes depending on how much time and money you have to spend.

Technical Summary: The Limits of Training Data Size in Foundation Models

Problem Statement
The paper addresses a fundamental tension in the scaling of foundation models: the trade-off between data quality and data quantity under a fixed training budget. Current estimates of the "data wall" (the point where human-written text is exhausted) rely on two independent multipliers: the fraction of raw text that survives quality filtering, and the number of times surviving tokens can be usefully repeated. The authors argue these factors are not independent. On a fixed pool of unique tokens ( $U$ ), filtering to a retention rate $f$ improves average quality but forces the model to traverse the data $1/f$ times. Since repeated tokens exhibit diminishing returns (exponential decay in value), aggressive filtering converts unique tokens into repeated ones, potentially incurring a cost that grows with the training budget ( $D$ ). The central empirical question is: given a fixed pool $U$ and budget $D$ , which retention fraction $f$ minimizes held-out loss, and how does this optimum shift as the ratio $D/U$ increases?

Methodology
The study employs a controlled experimental design to isolate the effects of filtering, scarcity, and repetition.

Data Pool: A fixed pool of 24.0 million tokens derived from the first shard of the FineWeb dataset.
Quality Filter: Documents are scored using the released FineWeb-Edu classifier (a linear regression head on an embedding model trained on "educational value").
Corpus Construction: Three families of corpora were built from the fixed pool:
1. Quality-filtered ( $f$ ): The top $f$ fraction of tokens by classifier score.
2. Random subsets ( $f$ ): Randomly selected tokens matching the size of the quality-filtered sets (serving as a control for scarcity).
3. Up-weighted mixtures: The full pool is retained, but the top $f$ slice is oversampled $k=1/f$ times (serving as a "soft" alternative to hard filtering).
Training Setup: Small decoder-only transformers (ranging from 1.3M to 16.9M parameters) were trained on these corpora. The main grid used a 7.34M parameter model.
Budgets: Training budgets of 6.1M, 12.3M, and 24.6M tokens were used, resulting in repetition counts ranging from 0.26 to 16.4 passes over the unique tokens, depending on the corpus size.
Evaluation: Models were evaluated on three held-out sets:
1. Web: Raw web text (the distribution the pool was drawn from).
2. Edu: High-scoring web text (overlapping with the filter's target).
3. WikiText-103: Independent, curated encyclopedic text.
Robustness: The study includes matched controls, multiple model sizes, and hyperparameter sweeps (learning rate and weight decay) to ensure results are not artifacts of specific recipes.

Key Results
The experiments yield three primary findings, all exceeding the measured run-to-run noise band of ~0.055 nats per token:

Target Dependence: The sign of the filtering effect depends entirely on the evaluation target.
- On the raw web target, filtering never helps; loss increases monotonically as the retention rate $f$ decreases. Aggressive filtering (keeping the top 1/16) costs up to 0.44 nats per token compared to keeping all data.
- On high-quality targets (Edu and WikiText-103), filtering can be harmless or mildly beneficial at low budgets, but the benefit is small and often sits at the edge of the noise band.
Budget Dependence: The optimal retention rate loosens as the budget grows.
- As the budget increases, the cost of repetition outweighs the benefit of quality. For example, on WikiText-103, the top-1/16 corpus was marginally better than the unfiltered pool at a 6.1M token budget (4.1 passes) but became significantly worse at a 12.3M token budget (8.2 passes). The loss-minimizing fraction shifts toward keeping more data as $D/U$ increases.
Hard Filtering vs. Up-weighting: The method of acting on quality scores matters.
- At aggressive retention rates (e.g., top 1/16), up-weighting (keeping all data and oversampling the high-scoring slice) strictly outperforms hard filtering (deleting the low-scoring slice) across all targets and budgets. The advantage of up-weighting reaches up to 0.39 nats.
- At mild retention rates (e.g., top 1/4), hard filtering and up-weighting are statistically indistinguishable.
- Random vs. Selection: On high-quality targets, score-based selection significantly outperforms random subsets of the same size, indicating the classifier captures a genuine distributional signal. However, on the raw web target, random subsets perform nearly as well as the full pool, suggesting the damage from filtering is primarily due to distribution shift rather than scarcity.

Significance and Claims
The paper does not claim to determine the optimal filtering strategy for frontier-scale models (which are orders of magnitude larger). Instead, it establishes a mechanism and an exchange rate within a specific regime.

Interdependence of Multipliers: The study demonstrates that the "quality multiplier" (fraction surviving filtering) and the "repetition multiplier" (epochs) are not independent. Filtering reduces the unique token count, thereby increasing the repetition count for a fixed budget. Treating them as independent variables systematically overestimates the value of a filtered corpus.
Practical Guidance: The authors suggest that thresholds tuned on data-abundant regimes (single-epoch ablations) are likely suboptimal when the pool binds. They recommend:
- Setting retention rates against the specific budget, not the corpus size.
- Using up-weighting rather than deletion for aggressive concentration on high-quality data.
- Verifying that filtering results hold across different target distributions, as a filter tuned for one distribution may be harmful to another.
Negative Result: The authors report that a design with only two budget points per corpus is insufficient to identify the repetition-decay constant ( $R^*_D$ ) separately for filtered and random data, cautioning against similar thin-grid studies in the literature.

In conclusion, the paper argues that on a fixed pool, aggressive filtering is a "lossy" way to act on quality scores compared to up-weighting, and that the optimal strategy shifts toward retaining more data as the training budget increases.

The Great Library Experiment

The Surprising Findings

The Bottom Line

Technical Summary: The Limits of Training Data Size in Foundation Models

More like this