💬 NLP

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

The paper introduces FineInstructions, a method for generating billions of synthetic instruction-response pairs from pre-training corpora that enables large language models to be pre-trained from scratch solely on an instruction-tuning objective, resulting in superior performance on free-form response benchmarks compared to standard pre-training approaches.

Original authors: Ajay Patel, Colin Raffel, Chris Callison-Burch

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ajay Patel, Colin Raffel, Chris Callison-Burch

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to teach a super-smart robot how to chat with humans. Usually, the way we do this is like giving the robot a massive library of books and saying, "Read everything and guess the next word." This is called pre-training. It's how the robot learns facts, history, and how sentences work. But here's the catch: after reading millions of books, the robot is great at finishing sentences but terrible at actually answering questions or following instructions. It's like a student who has memorized the entire encyclopedia but freezes when you ask, "What's the best pizza place near me?"

To fix this, we usually give the robot a tiny, special set of "instruction cards"—questions and perfect answers—and say, "Learn this style!" But there's a problem: we don't have enough of these cards. We only have a few thousand or maybe a few million, which is like trying to teach a whole school of students with just one textbook.

The Big Idea: The "Recipe Book" Revolution
The authors of this paper, published at COLM 2026, came up with a wild idea: Why not turn the entire library of books into instruction cards?

They built a machine called FineInstructions. Think of it as a magical kitchen.

The Ingredients: They took a huge pile of real documents from the internet (about 300 billion words worth of text).
The Recipe Cards: They started with about 18 million real questions and prompts that actual humans have asked online (like "Which is more influential: Messi or Ronaldo?"). They turned these into "recipe cards" (templates) by replacing the specific names with blanks. For example, "Between and , which is more ?"
The Cooking: The machine looks at a document (say, a blog post about smartwatches) and asks, "Does this blog post fit any of our recipe cards?" If the blog talks about an Apple Watch and a Garmin, it matches the "Which is more rugged?" recipe.
The Meal: The machine then fills in the blanks using the blog post to create a perfect question and answer pair. It's like the blog post suddenly became a Q&A session.

The Result: A Billion New Lessons
By doing this, they created a dataset called FineInstructions containing over 1 billion synthetic instruction-answer pairs. This is huge! It's like turning the entire library into a billion flashcards.

What They Found (The Proof)
The team trained small robot brains (models with 1.8 billion parameters) from scratch using only these new instruction cards. They didn't use the old "guess the next word" method at all.

The Score: When they tested these robots on benchmarks that measure how well they answer real human questions (like MixEval, MT-Bench-101, and AlpacaEval), the robots trained on FineInstructions crushed the competition.
The Comparison: They compared their method against other fancy ways of trying to turn books into questions (like the IPT and Nemotron-CC methods). The FineInstructions robots won by a significant margin. For instance, on the MixEval benchmark, they improved performance by about 39% compared to the Nemotron-CC method and 69% compared to standard training.
The "Win Rate": When pitted against other models in a head-to-head chat, the FineInstructions model won about 76% of the time against the Nemotron-CC baseline.

What They Explicitly Say It's NOT
It's important to know what this paper doesn't claim.

It's not about "Perplexity": The authors admit that if you measure the robot's ability to predict the next word in a sentence (a standard test called perplexity), their method might actually be worse than the old way. They don't care about that test. They care about the robot being useful to humans.
It's not "Magic" or "Distillation": Some previous methods tried to use a super-smart robot to write all the questions. The authors argue that this just makes the new robot copy the old one's style without really learning new things. Their method is different because it uses real human questions and grounds the answers in real documents, not made-up stories.
It's not a "Solved Problem": The paper suggests that this approach is a major step forward, but they note that complex templates are still hard to match perfectly. They also admit that their current setup uses some manual tuning (like picking a specific matching score of 0.865) that might not be the absolute perfect setting yet.

The Bottom Line
The paper suggests that by restructuring how we feed data to robots—turning raw text into a billion realistic Q&A pairs—we can teach them to be much better at following instructions and helping humans, even if we have to skip the traditional "guess the next word" training. It's a shift from teaching a robot to be a "reader" to teaching it to be a "helper."

The authors are confident in their results because they ran controlled experiments where every robot got the exact same amount of text to learn from, just in a different format. The data shows that the "instruction-only" format leads to smarter, more helpful robots for real-world tasks.

Technical Summary: FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

Problem Statement

Large Language Models (LLMs) are typically pre-trained using self-supervised next-token prediction on vast, unstructured text corpora. While this stage imparts the majority of a model's knowledge, it is often followed by a separate "instruction-tuning" phase using a relatively small amount of supervised instruction-answer data to align the model with user intent. Existing instruction-tuning datasets suffer from limitations: they are often small (thousands of examples), narrow (focused on academic NLP tasks), or rely on distillation from frontier models, which may only superficially mimic capabilities. Consequently, the pre-training stage remains responsible for encoding knowledge, while the instruction-tuning stage is largely limited to learning response styles. Furthermore, standard pre-training often wastes compute on low-quality noise (headers, footers) and does not align well with the downstream distribution of user prompts.

Methodology: The FineInstructions Pipeline

The authors propose FineInstructions, a procedure to transform internet-scale pre-training documents into billions of synthetic instruction-answer pairs, effectively enabling supervised instruction-tuning at pre-training scale. The pipeline consists of eight key stages:

Instruction Template Generation:
- Source: The authors collected ~21.5M real user-written queries from diverse sources (e.g., WildChat, LMSys, StackExchange, Reddit QA, GooAQ).
- Genericization: Using a "Query Genericizer Model" (a distilled Llama-3.2 1B model), these queries are converted into ~18M generic instruction templates. Specific entities or scenarios in the queries are replaced with <fi> tags containing natural language descriptions of the required content (e.g., <fi>Entity A</fi>).
- Filtering: Harmful queries are removed, and data is decontaminated from common benchmarks.
Document Matching:
- Embedding: A semantic similarity model (BGE-M3) is used to embed "compatible document descriptions" for the templates and the pre-training documents themselves.
- Gaussian Pooling: To ensure templates match specific sections of long documents, the authors introduce a custom "Gaussian pooling" layer. Instead of a single global embedding, the model generates $K=5$ local embeddings representing distinct chunks of the document, plus a global embedding.
- Retrieval: For each document, the system retrieves compatible instruction templates based on cosine similarity (threshold 0.865). A weighted random sampling strategy ensures diversity in template complexity.
Instantiation and Answer Generation:
- Instantiator Model: A distilled Llama-3.2 3B model is trained to instantiate the retrieved templates using the document content and generate answers.
- Grounding: To avoid hallucination and model collapse, answers are primarily grounded in the source document. The model is instructed to copy excerpts directly. A special tagging mechanism (<excerpt>...</excerpt>) is used to reduce token generation costs by allowing the model to generate tags with ellipsis notation for long spans, which are expanded programmatically later.
- Constraints: The ratio of excerpted text in generated answers is enforced to be $\ge 0.80$ .
Judging and Filtering:
- A "Flow Judge" model (a 3.8B parameter distilled model) evaluates the generated instruction-answer pairs on a 5-point Likert scale.
- Pairs scoring $\ge 4$ (addressing the query without extraneous content) are retained. This stage also filters out malformed instantiations and answers not grounded in the source.
Training:
- The resulting dataset, FineInstructions, contains over 1 billion synthetic instruction-answer pairs.
- Models are pre-trained from scratch solely on this instruction-tuning objective (Instruction: {{instruction}} \n\n Answer: {{answer}}), rather than standard next-token prediction.

Key Contributions

Large-Scale Template Creation: The creation of ~18M instruction templates derived from real user queries, enabling the generation of highly diverse and in-distribution synthetic data.
The FineInstructions Procedure: A scalable pipeline for converting unstructured pre-training documents into supervised instruction-answer pairs, utilizing efficient distilled models for genericization, matching, and instantiation.
Empirical Validation: Demonstration that pre-training solely on FineInstructions outperforms standard pre-training and other synthetic transformation baselines (Instruction Pre-Training, Nemotron-CC, WRAP) on benchmarks correlated with human judgment of response quality.
Open Resources: Release of code, trained models, and the FineInstructions dataset (1B+ pairs).

Results

The authors conducted controlled token-for-token training experiments using 1.8B parameter models on two corpus scales (IPT ~23B tokens and Nemotron-CC ~300B tokens).

Performance Gains: FineInstructions significantly outperformed baselines on MixEval (a benchmark correlating with human judgment), MT-Bench-101 (multi-turn subjective queries), and AlpacaEval (head-to-head win rates).
- On the IPT corpus, FineInstructions showed a ~69% relative improvement on MixEval compared to standard pre-training.
- On the Nemotron-CC corpus, it showed a ~39% improvement on MixEval.
- In AlpacaEval, outputs from FineInstructions were consistently preferred over all other pipelines.
Generalization: Improvements were consistent across both knowledge-focused tasks (MixEval) and open-ended real-world tasks (MT-Bench-101, AlpacaEval).
Efficiency: Experiments at different model scales (300M, 1.8B, 7B) indicated that FineInstructions allows smaller models to reach performance levels comparable to or exceeding larger models trained on standard baselines, given equivalent token and compute budgets.

Significance and Claims

The paper claims that restructuring pre-training data into instruction-answer formats improves knowledge absorption efficiency and aligns the pre-training distribution more closely with downstream user usage. By transforming the learning objective from "predict the next word" to "respond to an instruction," the model learns to follow instructions and generalize across tasks during the pre-training phase itself, rather than relying on a separate, data-scarce instruction-tuning stage.

The authors explicitly note that their approach is optimized for downstream user utility (producing useful responses) rather than intermediate language modeling proxies like perplexity. They concede that standard pre-training likely outperforms FineInstructions on perplexity-based metrics, as FineInstructions does not train on raw next-token prediction over documents. However, the proposed method offers a more effective path for training models capable of high-quality instruction following and reasoning at scale.

The work also highlights that the diversity of the generated instructions (derived from real user queries) is a key differentiator from prior synthetic methods that often focus on narrow academic task formats. The pipeline effectively mitigates the risk of model collapse by grounding answers in source documents and filtering for quality, while the use of distilled models ensures the procedure is computationally feasible at pre-training scales.