💬 NLP

Data Repetition Beats Data Scaling in Long-CoT Supervised Fine-Tuning

This paper demonstrates that, contrary to standard intuition, supervised fine-tuning on chain-of-thought data achieves superior reasoning performance by repeatedly training on smaller datasets until full memorization is reached, rather than by scaling up the dataset size for a single epoch.

Original authors: Dawid J. Kopiczko, Sagar Vaze, Tijmen Blankevoort, Yuki M. Asano

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dawid J. Kopiczko, Sagar Vaze, Tijmen Blankevoort, Yuki M. Asano

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but distracted student how to solve incredibly hard puzzles, like complex math problems or deep science questions. In the world of artificial intelligence, this student is a "language model," a computer program that learns by reading text. To make these models good at reasoning, scientists use a process called "Supervised Fine-Tuning" (SFT). Think of this as giving the student a stack of example solutions written by a master teacher. The student reads these examples and tries to copy the thinking process, step-by-step, until they can solve similar problems on their own.

For a long time, the golden rule in machine learning has been "more is better." The intuition is simple: if you want a student to learn well, you should give them a massive library of unique books. The more different examples they see, the better they should get at generalizing to new situations. It's like believing that reading 10,000 different stories is always better than reading the same 100 stories over and over. But what if that rule is wrong? What if, for this specific type of "reasoning" training, the secret isn't a bigger library, but a deeper understanding of a smaller one? This is the question a new paper from the COLM 2026 conference tackles, challenging our basic assumptions about how AI learns to think.

The researchers behind this study decided to test the "more is better" rule by running a series of controlled experiments. They took a standard AI model and trained it on "Chain-of-Thought" data—long, detailed reasoning traces where the model learns to think through a problem before answering. They set up a strict budget: a fixed amount of "computing effort" (like a fixed number of hours the student is allowed to study). They then compared two strategies. Strategy A was to give the student a huge pile of unique examples (51,200 samples) but let them read through the whole pile only once. Strategy B was to give the student a tiny pile of examples (as few as 200 or 400 samples) but let them read through that same small pile many, many times—up to 128 times.

The results were surprising and counterintuitive. The paper found that the student who read the small pile over and over (Strategy B) became significantly better at solving new, difficult problems than the student who rushed through the massive library just once. On tough benchmarks like the AIME math competitions and the GPQA science quiz, the model trained on just 400 samples for 128 epochs (repetitions) outperformed the model trained on 51,200 samples for just 1 epoch by a massive margin—improving accuracy by 12 to 26 percentage points. It's as if the student who memorized a few key stories perfectly could solve new mysteries better than the student who skimmed a thousand different books.

The paper also explored why this happens and what the limits are. They discovered that the improvement didn't come from learning new facts, but from the model "memorizing" the structure of the reasoning. As the model repeated the small dataset, it eventually reached a point where it could predict the next word in the training examples with near-perfect accuracy (100%). Once the model hit this "full memorization" of the training data, its performance on new tests stopped getting better and plateaued. This suggests that for this specific type of training, the goal isn't to see as many different examples as possible, but to internalize the pattern of reasoning until it becomes second nature.

Interestingly, the researchers checked if this "repetition" caused the model to forget everything else it knew (a problem called "catastrophic forgetting"). They found that repeating the small dataset actually caused less forgetting than reading the huge dataset once. It seems that by deeply practicing the reasoning structure, the model became more confident in its own logic, which helped it stick to the task without losing its general knowledge.

However, the authors are careful not to call this a magic bullet that solves everything. They explicitly rule out the idea that this works for all data; for instance, if the teacher providing the examples is weak or the examples are wrong, the benefits change. They also note that while the "repetition advantage" is robust, the exact reason why memorizing the training data helps generalization is still a mystery. They suggest that the model might be unlocking hidden capabilities it already had, rather than learning something entirely new.

In short, this paper suggests that for teaching AI to reason, we might be wasting our time trying to find bigger and bigger datasets. Instead, we might get better results by taking a smaller, high-quality set of reasoning examples and letting the model study them until it knows them inside out. The authors propose a new practical rule: keep training on the small dataset until the model can perfectly predict the training text, and then stop. While they haven't fully explained the "why" behind this phenomenon, they have shown that sometimes, in the world of AI, reading the same page 100 times is far more powerful than skimming 100 different pages.

Technical Summary: Data Repetition Beats Data Scaling in Long-CoT Supervised Fine-Tuning

Problem Statement

Standard machine learning intuition posits that training with more unique samples yields better generalization, a principle that has driven the scaling of unique corpora in large language model (LLM) pretraining and post-training. However, high-quality long Chain-of-Thought (CoT) demonstrations for reasoning models are scarce, often requiring expensive human annotation or careful distillation from larger models. This scarcity raises a critical question: under a fixed computational budget (total gradient updates), is it more effective to train on a large dataset for a single epoch, or to train on a smaller dataset for multiple epochs? The paper challenges the prevailing assumption that data diversity is always superior, investigating whether data repetition can substitute for data scaling in the supervised fine-tuning (SFT) stage of reasoning models.

Methodology

The authors conduct controlled experiments to isolate the effects of data repetition versus data scaling while holding the total update budget constant.

Models: Experiments utilize base checkpoints of three recent language models: Olmo3-7B, Qwen3-8B, and Qwen3-4B. These models are used prior to instruction tuning to ensure a clean starting point for SFT dynamics.
Datasets: The primary dataset is "Dolci SFT 7B," a collection of distilled long-CoT demonstrations covering math, coding, and instruction following. The authors create nested training splits ranging from 200 to 51,200 samples, ensuring smaller sets are subsets of larger ones. Additional experiments use math-specific data distilled from Qwen3-0.6B and Qwen3-8B teachers, partitioned by the correctness of the final answer (positive vs. negative trajectories).
Experimental Grid: The core experimental design varies the number of unique samples ( $N$ ) and the number of training epochs ( $E$ ) such that the product $N \times E$ (the update budget) remains constant. For example, a budget of 51,200 updates is achieved by training on 51,200 samples for 1 epoch, 3,200 samples for 16 epochs, or 200 samples for 256 epochs.
Evaluation: Models are evaluated on three challenging reasoning benchmarks: AIME 2024, AIME 2025, and GPQA. Metrics include:
- Accuracy@n: Average accuracy over $n$ independent generations.
- Pass@n: The fraction of problems solved in at least one of $n$ attempts.
- Termination Rate: The fraction of generations that successfully conclude with an end-of-sequence token.
- Catastrophic Forgetting: Measured via MMLU performance to assess retention of general knowledge.
Training Dynamics: The study tracks training token accuracy (memorization), validation loss, and prediction entropy to understand the convergence behavior.

Key Results

1. The Repetition Advantage

Contrary to standard scaling laws, the experiments demonstrate that under a fixed update budget, training for more epochs on smaller datasets substantially outperforms training on larger datasets for fewer epochs.

Performance Gains: On the Olmo3-7B model with a 51,200 update budget, training for 128 epochs on 400 samples outperformed 1 epoch on 51,200 samples by 12–26 percentage points on AIME and GPQA benchmarks.
Consistency: This trend holds across different model sizes (Olmo3-7B, Qwen3-8B, Qwen3-4B), different benchmarks, and different data sources (general Dolci dataset vs. math-specific distillations).
Saturation Point: Performance gains from additional epochs begin to plateau around 32–64 epochs, suggesting a saturation point where further repetition yields diminishing returns.

2. Memorization as a Stopping Criterion

The authors identify a strong correlation between training set memorization and downstream performance.

Token Accuracy: As epoch count increases, the model's accuracy on the training set tokens approaches 100% (full memorization).
Convergence Signal: Downstream performance improvements plateau precisely when the model achieves near-perfect token-level accuracy on the training data. This suggests that training token accuracy can serve as a practical, data-independent stopping criterion for epoch scaling, replacing the need for expensive, undirected data scaling.

3. Termination and Reasoning Structure

A strong correlation exists between the termination rate and accuracy. Single-epoch models often fail to terminate (24% termination rate), whereas multi-epoch models (32 epochs) achieve high termination rates (89%). This indicates that repetition helps models internalize not just the reasoning patterns but also the structural convention of concluding long reasoning chains.

4. Robustness to Data Quality and Errors

Teacher Quality: The repetition advantage persists regardless of whether the distilled data comes from a weak (0.6B) or strong (8B) teacher model. However, with a weak teacher, increasing the total update budget (scaling data) can lead to performance degradation, echoing "weak-to-strong" generalization issues.
Negative Trajectories: Surprisingly, training on incorrect reasoning traces (negative trajectories) does not degrade performance and, in some cases (AIME'24, GPQA), yields performance comparable to or slightly better than training on correct traces. This suggests that exposure to difficult reasoning attempts, even when flawed, benefits the student model.

5. Catastrophic Forgetting

While multi-epoch training on small datasets leads to overfitting (rising validation loss, decreasing entropy), it does not cause additional catastrophic forgetting compared to single-epoch training on large datasets. In fact, epoch scaling results in less degradation on the MMLU benchmark than data scaling, offering a strictly better trade-off between reasoning performance and knowledge retention.

Significance and Claims

The paper posits the "repetition advantage" as a counter-intuitive phenomenon where full memorization of a small dataset coincides with improved generalization in long-CoT SFT.

Practical Heuristic: The authors propose a practical approach for reasoning SFT: scale epochs until training token accuracy saturates, rather than scaling unique data. This offers a cost-effective alternative to acquiring massive amounts of unique CoT data.
Open Problem: The paper explicitly frames the explanation of why memorization under repetition improves generalization in this specific context as a novel, open problem for the community. The authors note that while they identify correlates (memorization, termination behavior), the definitive causal mechanism remains to be understood.
Paradigm Shift: The findings challenge the default industry practice of single-epoch training on the largest available dataset, suggesting that epoch count and dataset size should be treated as first-class decision variables in reasoning SFT.

The paper concludes that for reasoning-focused SFT, "more unique samples" is not always better; under fixed compute, "more epochs" on smaller, high-quality datasets is a superior strategy.