💬 NLP

How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

This paper presents a systematic study identifying that structured output formats and source data selection are critical for high-quality synthetic pretraining, leading to the creation of **FinePhrase**, a cost-effective 486-billion-token dataset that outperforms existing baselines while demonstrating that generator models larger than 1B parameters offer no additional benefit.

Original authors: Joel Niklaus, Atsuki Yamaguchi, Michal Štefánik, Guilherme Penedo, Hynek Kydlíček, Elie Bakouch, Lewis Tunstall, Edward Emanuel Beeching, Thibaud Frere, Colin Raffel, Leandro von Werra, Thomas Wolf

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Joel Niklaus, Atsuki Yamaguchi, Michal Štefánik, Guilherme Penedo, Hynek Kydlíček, Elie Bakouch, Lewis Tunstall, Edward Emanuel Beeching, Thibaud Frere, Colin Raffel, Leandro von Werra, Thomas Wolf

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but inexperienced student (a Large Language Model) how to become a world-class expert. You have a massive library of books (the internet), but many of them are messy, full of typos, or written in confusing slang. You need to clean them up and present the information in a way that helps the student learn the fastest.

For a long time, researchers thought the only way to do this was to find the "perfect" books. But the internet is running out of high-quality text. So, researchers started using AI to rewrite the messy books into better versions. This is called Synthetic Data.

However, until now, nobody knew how to rewrite them best. Should the AI act like a strict teacher? A chatty friend? Should it use a super-smart (and expensive) AI to do the rewriting, or is a smaller, cheaper AI good enough?

The paper "How Can We Synthesize High-Quality Pretraining Data?" by the Hugging Face team is like a massive cooking competition where they tested every possible recipe to find the perfect dish. They generated over one trillion tokens (words) to figure out the secret sauce.

Here is what they discovered, explained simply:

1. The "Format" is More Important Than the "Chef"

The Analogy: Imagine you are teaching a child math.

Option A: You give them a messy paragraph of text that contains a math problem.
Option B: You rewrite that text into a clear, step-by-step math word problem with a solution.
Option C: You turn it into a FAQ (Frequently Asked Questions) or a Table.

The Finding: The researchers found that Option B, C, and D (the structured formats) were magic. When they forced the AI to rewrite web text into Math problems, FAQs, Tables, or Tutorials, the student learned much faster.

Why? It's like turning a messy pile of Lego bricks into a clear instruction manual. The structure helps the brain (the model) understand the logic better than just reading a wall of text.
The Surprise: They thought they needed a "Super Chef" (a massive, expensive AI model) to write these instructions. They were wrong. A small, cheap "Junior Chef" (a 1.7 Billion parameter model) did just as good a job as the "Master Chef" (a 27 Billion parameter model). In fact, the bigger chefs sometimes got too rigid and repetitive, which actually hurt the learning.

2. Don't Throw Away the "Real" Stuff

The Analogy: Imagine you are making a smoothie.

Synthetic Data is like a high-tech, flavor-enhanced powder.
Real Web Data is like fresh, real fruit.

The Finding: If you make a smoothie only out of the powder (synthetic data), it tastes weird and the student gets confused about how real people talk. The model starts to lose its "common sense."

The Solution: You must mix the powder with the real fruit. The best results came from mixing the rewritten synthetic data with original, high-quality web text. The synthetic data teaches logic and facts, while the real data teaches the model how to speak naturally and understand human nuance.

3. Quality of the "Raw Ingredients" Matters Less Than You Think

The Analogy: You have a pile of slightly bruised apples (low-quality web text) and a pile of perfect apples (high-quality text).

Old Belief: You must use the perfect apples to make a great pie.
New Finding: If you have a great recipe (the structured prompts) and you mix the pie with a little bit of perfect apple sauce (high-quality mix-in data), you can actually use the bruised apples and still make a delicious pie!
Why? The rewriting process "up-cycles" the bad data. It takes the messy information and forces it into a clean, logical format. As long as you mix it with some high-quality data, the source of the original text doesn't matter as much.

4. The "Template Collapse" Trap

The Analogy: Imagine a robot teacher who is too perfect. Every time it explains a concept, it uses the exact same sentence structure, the exact same font, and the exact same tone.

The Problem: The student gets bored and stops learning because everything feels the same. This is called "Template Collapse."
The Finding: The researchers found that the "Junior Chef" (SmolLM2) was actually better because it was a little bit messy and varied. It didn't follow the rules too strictly. This variety kept the student's brain engaged. The "Master Chef" (Qwen 3) was too perfect and repetitive, which actually made the training less effective.

The Result: FINEPHRASE

Using these discoveries, the team built a new dataset called FINEPHRASE.

It contains 486 billion tokens of rephrased web text.
It was created using a small, cheap model (saving them massive amounts of money).
It is 30 times cheaper to produce than previous methods.
Performance: When they trained a model on this data, it beat all the previous "synthetic" records. It learned facts and logic better than anyone expected, while still keeping its natural language skills.

The Big Takeaway

To build the next generation of AI, you don't need to spend billions of dollars on the biggest supercomputers to rewrite data. Instead, you need better recipes.

Rewrite messy text into clear, structured formats (like Math, Tables, and FAQs).
Mix this new data with real, high-quality human text.
Use a small, efficient AI to do the rewriting, and let it be a little bit creative rather than robotic.

It's a shift from "bigger is better" to "smarter structure is better."

1. Problem Statement

The landscape of Large Language Model (LLM) pretraining is shifting from a focus on data quantity to data quality due to the plateau of high-quality, crawlable web text. While synthetic data (rephrasing existing text using LLMs) has emerged as a solution to expand training corpora, the field lacks a systematic understanding of how to generate it effectively. Prior work has explored isolated strategies (e.g., specific rewriting styles or large generator models) without comparing them across a unified framework. Key unanswered questions include:

Which rephrasing strategies yield the most effective training signals?
Does increasing the size of the generator model (e.g., from 1B to 70B parameters) improve the quality of synthetic data?
How do source data quality and the mix-in of original web data affect downstream performance?
What is the optimal balance between cost and performance?

2. Methodology

The authors conducted a systematic, controlled ablation study involving the generation of over one trillion tokens to evaluate three primary axes of the synthetic data pipeline:

A. Rephrasing Strategy (Prompt Design)

The study compared established prompts from prior work (e.g., Nemotron-HQ-Synth, REWIRE, BeyondWeb) against four new pedagogical structured formats:

Math: Converting text into mathematical word problems with step-by-step solutions.
FAQ: Transforming content into logical, self-contained Question-Answer pairs.
Table: Organizing information into structured Markdown tables followed by a QA pair.
Tutorial: Rewriting content as clear, step-by-step instructional guides.

B. Generator Model

The authors evaluated the influence of the generator model ( $G$ ) across two dimensions:

Parameter Scale: Tested models ranging from 135M to 27B parameters (using Gemma 3 and SmolLM2 families).
Architectural Families: Compared six distinct open-source families at a fixed ~1B scale (Gemma 3, Llama 3.2, Qwen 3, Granite 3.1, Falcon 3, SmolLM2).

C. Data Composition

The study analyzed the interaction between:

Source Data ( $D_{source}$ ): The raw text being rephrased (ranging from high-quality FineWeb subsets to low-quality/noisy web text).
Mix-in Data ( $D_{mix}$ ): The original web tokens mixed with synthetic data during the pretraining of the target model.
Training Regime: Models were trained from scratch on a 50/50 mixture of synthetic and original tokens (or synthetic-only) for 21B tokens.

D. Evaluation

Performance was measured by training a 1.2B parameter Qwen 2 architecture on the generated data and evaluating it across 12 benchmarks covering General Knowledge, Reading Comprehension, Reasoning, Natural Language Understanding (NLU), Math, and Table Understanding.

3. Key Findings & Results

A. Rephrasing Strategy is the Dominant Factor

Structured Pedagogical Formats Outperform: The four new formats (Math, FAQ, Table, Tutorial) consistently outperformed all existing baselines and the raw DCLM dataset.
- Math achieved the highest macro-average score (15.31), surpassing the DCLM baseline (13.77) by +1.54 points.
- Table and FAQ also showed significant gains (+1.06 and +0.68, respectively).
Simple Paraphrasing Fails: Strategies like "Distill," "Wikipedia," and "Knowledge List" provided negligible or negative utility compared to curated web text.
Diversity > Consistency: Qualitative analysis revealed that "template collapse" (extreme formatting consistency) harms performance. Models that produced diverse output lengths and structures (despite occasional instruction adherence failures) yielded better downstream results than highly consistent but repetitive outputs.

B. Generator Model Size: Diminishing Returns

No Benefit Beyond 1B Parameters: Increasing the generator size beyond 1B parameters provided no consistent performance gains. In many cases, larger models (e.g., 27B) performed worse than 1B models, likely due to over-optimization of formatting at the expense of content diversity.
Small Models are Sufficient: A 1.7B parameter model (SmolLM2) was found to be the optimal generator, offering the best cost-to-performance ratio.
Architecture Matters: The SmolLM2 family outperformed other ~1B models (like Llama 3.2 or Qwen 3) significantly, attributed to its instruction-tuning data which included explicit rewrite tasks.

C. Data Composition: The Mix-in is Critical

Necessity of Mixing: Training exclusively on synthetic data resulted in suboptimal performance, particularly in Natural Language Understanding (NLU) tasks (e.g., HellaSwag, PIQA). A mixture of synthetic and original web data is essential to preserve commonsense reasoning and linguistic diversity.
Mix-in Quality > Source Quality: The selection of the mix-in dataset ( $D_{mix}$ ) was a stronger driver of performance than the quality of the source text ( $D_{source}$ $D_{so u r ce}$ ).
- Using high-quality mix-in data (e.g., FineWeb-HQ) allowed the pipeline to successfully "up-cycle" low-quality source text, turning noisy web data into high-utility training tokens.

4. Key Contributions

FINEPHRASE Dataset: The authors released FINEPHRASE, a 486-billion-token open dataset of rephrased web text. It utilizes the optimal configuration identified in the study:
- Generator: SmolLM2 1.7B.
- Prompts: Math, FAQ, Table, Tutorial.
- Source: FineWeb.
- Mix-in: FineWeb-HQ.
Cost Efficiency: FINEPHRASE achieves 30x lower generation costs compared to previous methods (e.g., REWIRE using Llama 3.3 70B) while generating a larger volume of tokens (486B vs. 400B) and achieving superior performance.
Systematic Framework: The paper provides a reproducible experimental framework that isolates prompt design, model scale, and data composition, offering clear guidelines for future synthetic data generation.

5. Significance and Implications

Paradigm Shift in Synthetic Data: The study challenges the assumption that larger generator models are necessary for high-quality synthetic data. Instead, it emphasizes prompt engineering and structured output formats as the primary levers for performance.
Scalability: By demonstrating that compact models (1B-2B) can generate data that outperforms methods using 70B+ models, the paper makes high-quality synthetic data generation accessible to researchers with limited compute resources.
Data Up-cycling: The finding that low-quality source data can be effectively up-cycled when paired with a robust mix-in dataset significantly expands the potential reservoir of training data for LLMs.
Open Science: The release of the dataset, generation code, and all prompts allows the community to replicate and build upon these findings, accelerating the development of efficient pretraining pipelines.

In conclusion, the paper establishes that structured pedagogical prompts combined with small, specialized generator models and a high-quality mix-in strategy constitute the optimal approach for synthesizing pretraining data, offering a path to high-performance LLMs at a fraction of the previous computational cost.