Technical Summary: Disentangling Fixed-Source Synthesis from Source Expansion in Synthetic Data Scaling

Problem Statement

Synthetic data has become a standard component in training large language models (LLMs), with downstream performance scaling predictably with dataset size. However, existing scaling studies often conflate two distinct routes for expanding synthetic datasets:

1. Source Expansion (SE): Enlarging the information sources by adding seed materials (e.g., new documents or questions) or switching to stronger generators.
2. Fixed-Source Synthesis (FSS): Holding the source (seed questions and teacher model) fixed while scaling the generation budget (i.e., generating more responses per seed).

Prior work predominantly focuses on SE, leaving the returns of FSS underexplored. This paper aims to isolate FSS to determine its scaling limits, derive a predictive scaling law for it, and compare its efficacy against SE and various synthesis protocols under matched total-sample budgets.

Methodology

The authors model the synthetic data pipeline as a triple $(Q, T, p)$, where $Q$ is a seed-question pool, $T$ is a teacher model, and $p$ is a synthesis protocol. They isolate FSS by fixing $(Q, T)$ and varying only the per-question response budget $r$ via Rejection Sampling (RS).

Experimental Setup:

• Domains: Mathematics and Physics (from SuperGPQA).
• Models: Primary teacher is DeepSeek-V3.1 ($T_{DS}$); primary student is Qwen2-7B-Instruct ($S_{Q7}$). Cross-pair checks use Qwen2.5-72B ($T_{Q72}$) and Llama-3.1-8B ($S_{L8}$).
• Baseline Protocol: RS with temperature 1.2, collecting up to 32 correct responses per seed question.
• Evaluation Metric: Mean@8 (the mean fraction of correct responses when sampling 8 student outputs at temperature 0.7).
• Scale: The study involved approximately 2 billion teacher-inference tokens and over 500 student fine-tuning runs.

Research Questions (RQs):

1. RQ1 (FSS Scaling): With $(Q, T)$ fixed, does FSS improve student accuracy, and how does the gain scale with $r$?
2. RQ2 (SE vs. FSS): At matched total-sample budgets, how do gains from SE (adding seeds) compare to FSS (increasing $r$)?
3. RQ3 (Protocol Efficacy): Within FSS, do advanced synthesis protocols (diversification, selection, rewriting) outperform plain RS?

Key Contributions and Results

1. FSS Scaling Law and Diminishing Returns (RQ1)

The authors derive a specific scaling form for FSS based on the "quanta view" of learning, where features are covered binary (learned or not). Unlike standard scaling laws where data size is the axis, FSS scales a fixed source.

• Derived Form: The performance $P(r)$ follows a power-law decay toward a source-determined ceiling:
  $$P(r) \approx P_{\infty} - A r^{-\alpha}$$
  Here, $P_{\infty}$ is the ceiling determined by the fixed source $(Q, T)$, not the intrinsic noise of the task.
• Findings: FSS consistently improves student accuracy, but gains diminish rapidly as $r$ increases. The curve saturates earlier for weaker teachers.
• Prediction: A form fitted on low budgets ($r \le 16$) successfully predicts performance at the held-out highest budget ($r=32$) for all teacher-student pairs.

2. Source Expansion Outperforms Fixed-Source Scaling (RQ2)

The study compares SE (increasing the number of seed questions $|Q|$) against FSS (increasing responses $r$) at matched total example counts ($B = |Q| \times r$).

• Findings: At small budgets, SE and FSS are comparable. However, as the budget grows, expanding the real seed question set (SE) significantly outperforms spending the same budget on more responses from fewer seeds.
• Question Synthesis: Synthesizing new questions from existing seeds (an FSS variant) yields no gain over plain RS when matched on total sample count. The apparent benefit of prompt synthesis in prior work is attributed to budget effects rather than synthesis quality.

3. Limited Efficacy of Synthesis Protocols (RQ3)

Under fixed $(Q, T)$, the authors evaluated various protocols against plain RS, including:

• Temperature variants (0.6, 0.001).
• Embedding-based diversity selection.
• Persona prompting.
• Judge-free filtering (admitting wrong traces).
• Trace-level repair (rewrite $\to$ check $\to$ fix).
• Findings: While these protocols successfully shifted their targeted intrinsic metrics (e.g., diversity, noise levels), none consistently improved student performance over plain RS across the budget sweep. The gains from protocol changes are negligible once the source is fixed.

4. Robustness on Heterogeneous Sources

Experiments on a mixed STEM dataset confirmed that the FSS bound persists even with heterogeneous sources. At matched budgets, increasing the coverage of real seed questions (SE) remains superior to increasing response counts on a smaller pool.

Significance and Claims

The paper claims to provide a controlled setting for evaluating synthetic data scaling by disentangling source expansion from within-source synthesis.

• Bounded Scaling: FSS is a bounded scaling axis. It improves performance but is limited by the "source ceiling" ($P_{\infty}$) defined by the seed pool and teacher strength.
• Protocol Evaluation: The fixed-source setting serves as a rigorous control for comparing synthesis protocols. The authors find that under fixed sources, protocol variations contribute little compared to the fundamental choice of source expansion.
• Resource Allocation: For practitioners, the results suggest that when scaling synthetic data, adding new, diverse seed questions (Source Expansion) is a more reliable strategy than simply increasing the generation budget on a fixed set of seeds or relying on complex synthesis protocols.

The authors conclude that previous attributions of gains to "synthetic data scaling" often conflated the benefits of source expansion with those of increased sampling or protocol refinement. By isolating FSS, they demonstrate that while it is a valid scaling route, it is inherently bounded and less effective than expanding the source support.