🤖 machine learning

A Bitter Lesson for Data Filtering

This paper challenges the conventional wisdom that data filtering is essential for large model pretraining, demonstrating through scaling studies in high-compute, data-scarce regimes that sufficiently trained large models actually benefit from including low-quality and distractor data rather than filtering them out.

Original authors: Christopher Mohri, John Duchi, Tatsunori Hashimoto

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Christopher Mohri, John Duchi, Tatsunori Hashimoto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "No Filter" Rule

Imagine you are trying to teach a super-smart student (a Large Language Model) everything about the world. The traditional way to do this is to act like a strict librarian. You go through a massive library (the internet/Common Crawl), throw away the messy, poorly written, or repetitive books, and only give the student the "best" 1% of the collection.

This paper argues that if you have enough time and energy (computing power) to teach the student for a very long time, you should actually stop being a librarian.

The authors found that if you let the student read everything—the good books, the bad books, the repetitive lists, and the random noise—they will eventually learn better than if you had filtered the books for them. In fact, the "junk" data turns out to be helpful, not harmful, for a sufficiently large and well-trained model.

The Experiments: The "Gym" Analogy

To test this, the researchers set up a series of experiments that act like a gym for AI models.

1. The "Clean vs. Messy" Diet
They took a huge pile of internet text (the "Pool") and created different versions of it:

The Pool: The raw, unfiltered internet text.
The Filters: Versions where they removed specific things, like non-English text, repetitive paragraphs, or words that aren't common English words. Some filters were very strict (keeping only 2% of the text), while others were loose.

They trained models of different sizes (from tiny to huge) on these different diets.

The Result: For small models or short training sessions, the "Clean Diet" (filtered data) worked better. It's like a small child learning better from a picture book than a dictionary.
The Twist: As they made the models bigger and trained them longer (more "compute"), the "Messy Diet" (the full, unfiltered Pool) started to win. The models trained on the full pool eventually surpassed the ones trained on the filtered data, even though the filtered data was "higher quality."

2. The "Junk Food" Injection
To see just how robust these models are, the researchers deliberately added "junk" to the diet:

Random Strings: Made-up words like "htb hqovl bwdws."
Shuffled Sentences: Taking real sentences and scrambling the word order (e.g., "Paris France of capital is The").

The Surprise: Even with this garbage mixed in, the large models didn't break. They learned to ignore the noise. In a strange twist, the models actually learned better from the shuffled sentences than from the clean data in some cases. Why? Because even when the words are scrambled, the model can still see that "Paris" and "France" appear together often, helping it learn the connection between them. The "junk" acted like a tough workout that made the model stronger.

The "Bitter Lesson"

The paper references a famous concept called "The Bitter Lesson" by Richard Sutton. The lesson is this: Human intuition often fails when we scale things up.

We humans think, "We must curate the data because it's too messy." But the paper suggests that as we get more powerful computers, the simple approach of "just feed it everything" beats our complicated, human-designed rules for cleaning the data. The model is smart enough to figure out what is useful and what is noise on its own, provided it has enough time to learn.

The Catch: It Costs a Lot of Energy

There is a major condition for this to work: Compute.

The paper calculates that for the "read everything" approach to beat the "filter everything" approach, you need a massive amount of computing power. They estimate that for the full Common Crawl dataset (240 trillion words), you might need about 1e+30 FLOPs (a unit of calculation) to make the unfiltered data the clear winner.

Think of it like this: If you have a small budget, hiring a professional editor to clean your data is the smart move. But if you have a billion-dollar budget, you can afford to hire a million interns to read every single page of the internet, and the sheer volume of reading will teach the AI more than the editor ever could.

The Conclusion

The paper concludes that for the future of giant AI models, we might need to stop trying to be perfect curators. Instead of spending huge resources trying to filter out "bad" data, we should focus on building bigger models and training them longer on the raw, messy, unfiltered internet. The models are surprisingly resilient; they can handle the noise and even use it to learn better.

In short: If you have enough power, the best filter is no filter at all.

Technical Summary: A Bitter Lesson for Data Filtering

Problem Statement

The standard paradigm for pretraining large language models (LLMs) involves rigorous data filtering, where text from sources like Common Crawl (CC) is curated to retain only "high-quality" subsets. While this approach is widely accepted as essential, particularly in compute-constrained regimes, it conflicts with scaling laws that suggest performance improves with ever-increasing data volumes. Filtering drastically reduces the available token budget (e.g., DCLM-Baseline retains only ~1% of CC), potentially limiting the performance of models trained under high-compute, data-scarce conditions.

This paper investigates the hypothesis that data filtering is necessary in the limit of large compute. Specifically, it asks whether training directly on the full, unfiltered Common Crawl pool can eventually outperform training on carefully curated, filtered subsets when models are sufficiently large and trained for sufficiently long.

Methodology

The authors employ a systematic scaling study to evaluate the optimal performance ( $L^\star$ ) achievable on different datasets, defined as the minimum loss attainable by varying model size ( $M$ ) and training steps ( $N$ ) without computational constraints.

Experimental Setup:
- Datasets: The primary dataset is a 240 trillion token subset of Common Crawl (DCLM-Pool). The authors create subsets ranging from 670 million to 10 billion tokens.
- Filters: They compare the unfiltered pool against five filtered variants:
  - English filter: Removes non-English text.
  - Repetition filter: Removes documents with excessive repetition.
  - Stop word filter: Ensures the presence of common English stop words.
  - RefinedWeb: A combination of the above plus similar filters.
  - DCLM-Baseline: A heavily filtered pipeline retaining ~2.1% of the original data.
- Models: Dense Llama-style transformers ranging from 15 million to 7 billion parameters.
- Metrics: Primary metrics are average negative log-likelihood (NLL) on C4, Fineweb-Edu, and Cosmopedia. Downstream benchmarks (ARC-Easy, PIQA, SocialIQA) are also evaluated.
Scaling Strategy:
- The study scales both model size and training steps (including multiple epochs) to observe the "crossover point" where the unfiltered pool outperforms filtered datasets.
- They introduce "junk data" experiments by injecting randomly generated strings and shuffled-word documents into the training pool to test model robustness to noise.
- Scaling laws are derived to predict the compute threshold (in FLOPs) required for the unfiltered pool to become optimal for specific pool sizes.
Theoretical Analysis:
- The paper presents a theoretical model based on low-rank matrix factorization with orthogonal inputs. This model demonstrates that if a model's rank (capacity) is sufficient to cover the number of distinct tasks (or data distributions), it can optimally represent the prediction problem without interference from "noise" tasks. Conversely, smaller models suffer from interference.

Key Results

1. The "No Filter" Advantage at Scale

The central finding is that the full unfiltered Common Crawl pool eventually outperforms all filtered variants when models are sufficiently large and trained for sufficient steps.

Small Models: In low-compute regimes (small models, few steps), filtered datasets (like DCLM-Baseline) often yield lower loss.
Large Models: As model size increases (e.g., 330M to 1B parameters), the performance gap closes and reverses. The unfiltered pool achieves the lowest loss (e.g., 3.37 NLL on the 1B model), surpassing even the most aggressive filters.
Pareto Frontier: When plotting loss against compute (FLOPs), the unfiltered pool transitions from the worst-performing dataset to the best as compute increases. Notably, some filtered datasets (like those filtered only for repetition) never appear on the Pareto frontier; there is always a better dataset at any given compute level.

2. Robustness to "Junk" Data

The study tests the limits of model robustness by injecting low-quality data:

Random Strings: Models trained on a mix of CC and randomly generated strings (up to 200% of the pool size) eventually match the performance of the clean pool as model size increases.
Shuffled Words: Surprisingly, models trained on documents with shuffled word orders (preserving only unigram distributions) not only recover performance but, in some cases (330M+ models), outperform the unfiltered pool after sufficient training steps. This suggests large models can extract useful signal (e.g., co-occurrence statistics) even from highly degraded data structures.

3. Scaling Laws and Compute Thresholds

The authors derive scaling laws to predict when the unfiltered pool becomes optimal for the full 240 trillion token DCLM-Pool.

The number of training steps (epochs) required for the unfiltered pool to beat RefinedWeb grows super-linearly with pool size.
However, the required compute for this crossover is highly linear. The authors estimate that for the full 240T token pool, the unfiltered approach becomes optimal at approximately $10^{30}$ FLOPs. While this exceeds current frontier estimates (~ $5 \times 10^{26}$ FLOPs), it is within the realm of near-future projections (e.g., $10^{29}$ FLOPs by 2030).

4. Model Degradation and Factuality

Distribution Shift: While models are robust to shuffled words in terms of average loss, they show degradation on the very first token of a document (where the distribution shift is most acute). However, the authors argue this is negligible for most use cases involving longer contexts.
Factuality: A preliminary analysis of MMLU-related documents in CC suggests that actively harmful, non-factual content (e.g., conspiracy theories refuting established facts) is relatively rare compared to supporting or neutral content. Theoretical analysis suggests filtering is only beneficial if the "noise" is highly prevalent and the filter has a high true-positive-to-false-positive ratio, which may not hold for the broad distribution of CC.

Significance and Claims

The paper argues that data filtering may suffer from the "Bitter Lesson" (Sutton, 2019): human-designed heuristics that perform well at small scales are eventually superseded by simple, scalable approaches that leverage increased compute.

Rejection of Filtering Necessity: The authors reject the hypothesis that data filtering is strictly necessary. They provide evidence that no existing filter is likely to improve upon training directly on Common Crawl given enough compute and model capacity.
Robustness of Large Models: Sufficiently large models are highly robust to irrelevant or "junk" data and can extract useful information from noisy sources, including shuffled text.
Future Implications: While filtering remains important in current compute-constrained regimes, the paper suggests that as compute budgets scale toward $10^{30}$ FLOPs, the optimal strategy may shift toward using the full, unfiltered data pool, rendering complex curation pipelines obsolete.

The authors conclude that while harmful data (e.g., non-factual content mimicking high-quality text) can theoretically exist, it is not prevalent enough in Common Crawl to justify the massive data loss incurred by aggressive filtering, especially when the alternative is simply training larger models for longer.