Technical Summary: Bridging Compute- and Data-Optimal Pretraining

1. Problem Statement

Classical pretraining scaling laws (e.g., Kaplan et al., 2020; Hoffmann et al., 2022) operate under the assumption that fresh pretraining data is unbounded and scales freely with compute. However, the field is transitioning into a regime where compute capacity grows exponentially while high-quality pretraining data remains finite and costly to curate. In this "data-constrained" regime, practitioners must spend additional compute on derived tokens ($D'$) generated from a fixed corpus ($D$) via strategies like multi-epoch repetition, paraphrasing, or distillation.

The central question addressed by this work is: For a model of size $N$ trained on a fixed corpus $D$, how much can additional compute (spent on derived tokens $D'$) reduce the pretraining loss? Existing laws do not adequately describe the trade-off between compute and data when the data supply is bounded.

2. Methodology

The Compute-Data (CD) Scaling Framework

The authors propose a unified framework, CD-scaling laws, which bridges the gap between:

1. Compute-optimal scaling ($L_{Chin}$): Where data is unbounded ($D' = 0$).
2. Data-optimal scaling ($L_D$): Where compute is unbounded on a fixed corpus ($D' \to \infty$).

The core of the framework is a token effectiveness function, $\eta_{strat} \in [0, 1]$, which quantifies the value of a derived token relative to a fresh token. The proposed scaling law is:

$$L(N, D, D') = E + \frac{A}{N^\alpha} + \frac{B}{(D + \eta_{strat} \cdot D')^\beta}$$

Where:

• $N$: Model parameters.
• $D$: Fresh tokens (seen once).
• $D'$: Derived tokens.
• $\eta_{strat}$: Effectiveness coefficient dependent on the expansion strategy (repetition or paraphrasing), model size $N$, and data availability.

Experimental Setup

• Architecture: OLMo3 (Olmo et al., 2025).
• Model Sizes: 14M to 600M parameters.
• Data: Dolma-3 corpus (150B tokens).
• Strategies:
  1. Multi-epoch repetition: Repeating the corpus for additional epochs.
  2. Paraphrasing: Rephrasing documents using SmolLM2-1.7B-Instruct with specific style mixtures (question, math, FAQ, table).
• Fitting: The authors fit the scaling law parameters by sweeping fresh data sizes ($D$), derived token budgets ($D'$), and hyperparameters (learning rate, weight decay). They derive $\eta_{emp}$ empirically by finding the value that reproduces observed losses given the fixed 1-epoch scaling constants.

Parametric Form of $\eta$

Empirical analysis reveals that $\eta$ is not constant but decays as a function of:

1. Expansion Ratio ($r = D'/D$): $\eta$ decreases as more derived data is added.
2. **Tokens-Per-Parameter ($TPP = D/N$):**$\eta$ decays faster when fresh data is scarce relative to model size.
3. Model Size ($N$): Larger models saturate faster.

The authors model the effective derived data ($\eta \cdot D'$) as saturating to a ceiling $R^*$. The functional form adopted is:
$$\eta = \frac{R^*}{r} \left(1 - e^{-r/R^*}\right)$$
Where the saturation ceiling $R^*$ follows a power law dependent on $N$ and $TPP$:
$$R^*(D, N) = K \cdot (D/N)^\rho \cdot N^\sigma$$

3. Key Results

1. Non-Constant Effectiveness

The effectiveness of derived tokens is far from constant. It depends jointly on model size, the $TPP$ ratio, and the amount of derived data. As the corpus expands ($D' \to \infty$), the effective token count $D_{eff} = D + \eta D'$ saturates.

2. Three Operational Regimes

The CD-scaling law partitions training into three distinct regimes based on the binding constraint:

• Compute-bound: Between the compute-optimal frontier ($L_{Chin}$) and the data-optimal frontier ($L_D$). Here, additional compute on $D'$ significantly reduces loss.
• Data-bound: Between $L_D$ and the model-bound floor ($L_N$). Derived tokens have saturated; only fresh data can reduce loss further.
• Model-bound: The asymptotic limit where both compute and data are saturated; only increasing $N$ helps.

The "compute-bound" region shrinks as model size ($N$) and $TPP$ increase, indicating diminishing returns for additional compute on larger models or those with abundant data.

3. Strategy Selection and Optimization

• Small Models ($N \lesssim 600M$) & Low Data Budgets: Paraphrasing is significantly more effective than repetition.
• Large Models ($N \gtrsim 190M$) & High Data Budgets: Repetition outperforms paraphrasing.
• Epoch Recommendations: The "4-epoch rule" (often cited for Chinchilla) holds only for medium-scale models ($\sim 3B$) near $1\times$ Chinchilla data scale. For larger models and higher $TPP$, the recommended number of epochs decreases.
• Pareto Frontier: The framework identifies the optimal compute-data allocation, generalizing Chinchilla's compute-only optimization into a joint compute-data Pareto frontier.

4. Downstream Transfer

Validation loss remains a sufficient statistic for downstream capability (measured by BPB on LM tasks and accuracy on benchmarks like LAMBADA and SQuAD) regardless of whether the model was trained with 1-epoch, repetition, or paraphrasing. All strategies collapse onto a single curve when plotted against validation loss.

4. Significance and Claims

The paper claims to provide a unified framework that recovers existing laws as special cases:

• Compute-optimal scaling is recovered when $D' = 0$.
• Data-constrained scaling (Muennighoff et al., 2025) is recovered as a special case where the saturation ceiling $R^*$ is constant.
• Data-optimal scaling (Kim et al., 2025) is recovered as the limit where $D' \to \infty$.

Practical Implications:

1. Resource Identification: Practitioners can determine if their training run is limited by compute, data, or model capacity.
2. Allocation Strategy: The law prescribes the optimal split of compute between fresh data and derived data, moving beyond the "Chinchilla" heuristic.
3. Strategy Guidance: It provides data-driven recommendations on when to switch from repetition to paraphrasing and how many epochs or paraphrasing passes are optimal before returns saturate.

5. Limitations

The authors note several limitations:

• Empirical Fitting: The function $\eta$ absorbs residuals from the 1-epoch fit, potentially pushing $\eta_{emp} > 1$.
• Hyperparameter Sensitivity: Only learning rate and weight decay were swept; other hyperparameters (batch size, sequence length) were fixed.
• Compute Cost: Establishing these laws required over 250k H100-hours.
• Scope: The study focuses on repetition and paraphrasing; other strategies like self-distillation or synthetic structured data remain unexplored.
• Scale: Experiments were conducted on small-to-medium models (up to 600M), though the law was validated to extrapolate to larger sizes.