💬 NLP

Train Once, Answer All: Many Pretraining Experiments for the Cost of One

This paper proposes a cost-effective method for conducting multiple controlled pretraining experiments simultaneously within a single training run, demonstrating that this approach can replicate and extend prior research on data contamination and model behavior with minimal impact on overall performance.

Original authors: Sebastian Bordt, Martin Pawelczyk

Published 2026-03-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sebastian Bordt, Martin Pawelczyk

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a scientist trying to understand how a giant, super-smart robot brain (a Large Language Model or LLM) learns. Usually, to test a specific idea—like "What happens if we feed the robot a secret code?" or "What if we hide a specific fact in its diet?"—you have to build a brand new robot from scratch, feed it a specific diet, and see what happens.

The problem? Building and feeding these robot brains is incredibly expensive. It costs millions of dollars in electricity and computer time. So, scientists usually only get to run one experiment per robot. If they want to test ten different ideas, they have to build ten robots. That's like trying to test ten different recipes for a cake by baking ten separate cakes, when you only have enough flour for one.

This paper proposes a brilliant new way to bake: "Train Once, Answer All."

The Big Idea: The "Super-Salad" Approach

Instead of baking ten separate cakes, the authors decided to make one giant, super-complex salad.

Imagine you are a chef training a robot to be a master chef.

The Old Way: You make one salad with just tomatoes to see how the robot tastes tomatoes. Then you make a second salad with just cucumbers. Then a third with just peppers. You need ten bowls and ten batches of ingredients.
The New Way: You take one giant bowl. You add the tomatoes, the cucumbers, the peppers, the dressing, and even a few secret spices all at the same time. You train the robot on this one mixed bowl.

The authors asked: Can we mix all these different "ingredients" (experiments) into one training run without the robot getting confused or the flavors mixing up in a bad way?

What Did They Do?

They took a standard robot brain (called OLMo-2) and, while it was learning, they secretly injected ten different experiments into its training data simultaneously. These experiments were like different "flavors" of data:

The Memory Test: Hiding specific facts to see if the robot memorized them.
The Poison Test: Sneaking in "poison" data to see if the robot could be tricked into saying weird things later.
The Watermark Test: Adding invisible digital ink to the data to track where it came from.
The Math Test: Feeding it extra math problems to see if it got smarter at solving them.
The Contamination Test: See if the robot "forgot" the answers to test questions it saw during training.

They ran this single, massive training session on 210 billion words of text.

The Surprising Results

Here is the magic part: It worked perfectly.

No Flavor Mixing: Even though they mixed all ten experiments together, the robot learned each one independently. If they tested the "Math" experiment, the robot was good at math. If they tested the "Poison" experiment, the robot showed the poison behavior. The experiments didn't ruin each other.
The Robot Didn't Notice: The robot's overall "personality" and general smarts didn't change. It was just as good at normal tasks as a robot trained without these experiments. The experiments were like adding a pinch of salt to a soup; you taste the salt, but the soup doesn't turn into a different dish.
Replicating the Past: They took results from five different previous scientific papers (which usually required separate, expensive robot training runs) and successfully reproduced all of them in this single run.

The "Taste Test" (Checking for Interactions)

The scientists were worried: What if the "Tomato" experiment accidentally changed how the robot tasted the "Cucumber" experiment?

To check this, they invented a new tool called CPDT (Continual Pretraining Dependence Testing). Think of this as a "flavor detector." They took a partially trained robot and gave it small doses of the experiments to see if they interfered with each other.

The Result: The flavors stayed separate. The experiments were independent. The "Tomato" didn't mess up the "Cucumber."

Why This Matters

This is a game-changer for science.

Save Money: Instead of spending $100,000 to run ten experiments, you might only spend $10,000 to run one.
More Science: Researchers can now ask many more questions at once. We can learn about privacy, safety, reasoning, and memory all in the same afternoon.
Collaboration: Different research teams can pool their resources. One team can add their "poison" experiment, another can add their "math" experiment, and they can all share the cost of the single robot training run.

The Bottom Line

The authors proved that you don't need to bake ten cakes to test ten recipes. You can mix them all into one giant, delicious experiment and get all the answers you need. It's a smarter, cheaper, and faster way to understand how AI learns.

In short: They turned a "one experiment at a time" world into a "do it all at once" world, saving massive amounts of money and time while getting the same (or better) scientific results.

1. Problem Statement

Controlled pretraining experiments are a gold standard for understanding the causal relationship between training data and Large Language Model (LLM) behavior (e.g., studying memorization, data contamination, or reasoning). However, these experiments are severely constrained by computational cost. Traditionally, researchers must train a model from scratch for every individual experiment to isolate variables. This makes rigorous scientific inquiry into specific data effects prohibitively expensive, often limiting research to single-case studies or preventing the replication of findings across different model scales.

2. Methodology

The authors propose a novel paradigm: simultaneous multi-experiment pretraining. Instead of training one model per experiment, they conduct multiple independent experiments within a single training run.

Experimental Setup: The authors trained the OLMo-2-1B model (and variants up to 2.7B parameters) from scratch on 210 billion tokens. During this single run, they simultaneously implemented 10 distinct experiments modifying the training data.
The 10 Experiments: The experiments covered diverse research questions, including:
- Replications: Benchmark contamination, memorization patterns, verbatim memorization, pretraining poisoning, and forgetting curves (replicating works by Bordt et al., Panda et al., Liu et al., Zhang et al., and Pagliardini et al.).
- Novel Investigations: Knowledge acquisition (dynamically adjusting data frequency), mathematical reasoning (synthetic problem insertion), and Gaussian watermarks for privacy auditing.
- Total Modification: The experiments collectively modified approximately 3.7 billion tokens (1.8% of the total training data).
Baseline: A control model (OLMo-2-1B) was trained with identical architecture, initialization, and data ordering, but without the experimental modifications, to isolate the causal effects.

3. Key Contributions

A. Feasibility of Multi-Experiment Training

The paper demonstrates that multiple independent interventions can be applied to a single training run without significantly degrading the model's overall performance or training dynamics. The modified model (OLMo-2-1B-Exp) achieved benchmark scores comparable to the baseline, proving that the "cost of one" training run can yield results for ten distinct research questions.

B. Continual Pretraining Dependence Testing (CPDT)

A critical concern in this approach is experiment interaction (i.e., whether Experiment A influences the outcome of Experiment B). To address this, the authors introduced CPDT, a novel method to test for dependencies before full pretraining:

Method: They take an intermediate checkpoint and perform short, high-intensity continual pretraining runs with individual experimental interventions. They measure the outcome of every experiment $i$ when trained on the data of experiment $j$ .
Result: They constructed a dependence matrix showing that for their specific setup, the experiments are largely independent. The interactions were negligible compared to the natural noise in evaluation metrics. This contrasts with language modeling benchmarks, where they confirmed significant dependencies (e.g., training on ARC-Easy improves ARC-Challenge scores).

C. Dynamic Control Algorithms

The paper introduces a control algorithm for Knowledge Acquisition. Instead of using a fixed data insertion rate, the algorithm dynamically adjusts the frequency of fictional knowledge in the training data based on the model's current likelihood of the knowledge. This ensures the model acquires the knowledge by the end of training, allowing researchers to empirically determine the "repetition threshold" for knowledge acquisition.

4. Key Results

Successful Replication: All five replicated experiments (contamination, memorization, poisoning, etc.) faithfully reproduced the conceptual results of prior works, validating the experimental design.
- Example: The model successfully learned to extract prompts when triggered (poisoning) and exhibited significant forgetting of benchmark contamination over time.
Novel Findings:
- Knowledge Acquisition: The dynamic control algorithm successfully guided the model to acquire specific facts, demonstrating that knowledge acquisition can be steered via data frequency control.
- Mathematical Reasoning: Exposure to synthetic math problems (0.09% of data) significantly improved reasoning performance and showed length generalization (solving problems with more steps than seen during training).
- Watermarking: Gaussian watermarks added to embeddings were successfully detected throughout training, validating their use for data provenance auditing. The study also observed a "recency bias," where later watermarks were easier to detect.
Model Scaling: The effects of the experiments were detectable even in smaller models (179M parameters), though the magnitude of the effect generally increased with model size (e.g., larger models showed stronger contamination effects).
Training Stability: The presence of 10 simultaneous experiments had a minimal impact on training dynamics. The training loss, validation loss, and weight norms of the experimental model tracked the baseline almost perfectly.

5. Significance and Impact

Democratization of Rigorous Research: This approach drastically reduces the compute budget required for scientific inquiry into LLMs. It allows research teams to pool resources and answer multiple questions simultaneously, making controlled pretraining experiments accessible to smaller labs.
Scientific Rigor: By validating that experiments can be run in parallel without interference (via CPDT), the paper provides a framework for more efficient and scalable empirical studies in AI safety, privacy, and capability.
Open Science: The authors released a Python package, model checkpoints, and datasets, enabling the community to reproduce these multi-experiment runs and extend the methodology to other models (e.g., OLMo-3).

In summary, the paper argues that performing multiple pretraining experiments within a single training run is practical and scientifically valid, offering a pathway to overcome the computational bottlenecks that currently limit the study of large language models.