Technical Summary: InfoLaw

Problem Statement

Training Large Language Models (LLMs) is increasingly constrained by the scarcity of high-quality data. While upweighting high-quality data in pretraining generally improves performance, data-constrained regimes necessitate repetition of these samples. Existing literature indicates that excessive repetition leads to diminishing returns and potential performance degradation. Furthermore, standard scaling laws, which typically relate model loss to compute or token count, fail to reliably extrapolate across different data mixture recipes or under conditions of repetition. This failure renders the selection of optimal data recipes at scale underdetermined, often requiring costly grid searches over data configurations.

Methodology

The paper introduces InfoLaw (Information Scaling Laws), a data-aware scaling framework designed to predict model loss based on consumed tokens, model size, data mixture weights, and repetition levels. The core theoretical premise is to model pretraining as an information accumulation process where:

1. Quality controls information density: Higher-quality data contributes more information per token.
2. Repetition induces scale-dependent diminishing returns: Repeated exposure yields progressively smaller information gains.

The InfoLaw Framework

The framework defines a metric called Information ($info$), which aggregates the knowledge learned from a training corpus. The calculation involves:

• Quality Density ($f_d$): A parameterized function where higher-quality buckets (lower index $d$) are assigned higher information density. The paper fits an exponential decay function $f_d(\theta) = e^{-\theta \cdot d}$.
• Learning Rate Decay ($\lambda(N)$): A rate parameter dependent on the model's non-embedding FLOPs per token ($N$). The paper models this relationship as a logarithmic function $\lambda(N) = a \cdot \ln(N) + b$, capturing the saturation of learning capacity as model scale increases.
• Repetition Factor ($R_d$): The average number of times tokens from a specific quality bucket are seen, calculated based on the target token count ($K$), source token count ($S$), and mixture weights ($w$).

The total information learned is formulated as:
$$info = \sum_{d} f_d M_d \log(K) \left(1 - e^{-\lambda(N) R_d / \log(K)}\right)$$
where $M_d$ is the number of unique tokens in bucket $d$.

Finally, the framework establishes a power-law relationship between the model's validation loss ($L$) and the calculated Information:
$$L = \alpha \cdot info^{-\beta}$$

Experimental Setup

To fit the parameters of InfoLaw, the authors constructed a dataset varying along three axes: scale, quality distribution, and repetition level.

• Data Source: 3.7T English tokens from Common Crawl, partitioned into six quality buckets based on classifier scores.
• Sampling: A "LayerMix" sampling function was used to construct training sets with varying mixture weights ($w$) and repetition levels (by adjusting source size $S$ relative to target size $K$).
• Training: 9 models ranging from 252M to 1.2B parameters were trained from scratch under a 3.6x overtraining ratio.
• Evaluation: Performance was measured as average perplexity across five downstream tasks (HellaSwag, ARC-E/ARC-C, MMLU, TriviaQA).

Key Contributions

1. Identification of Scaling Law Failure: The paper demonstrates that traditional scaling laws (Loss vs. Compute) systematically mis-extrapolate under data repetition, yielding overly optimistic predictions for larger models.
2. The InfoLaw Formulation: A novel framework that unifies diverse data recipes and model scales onto a single power-law curve by replacing "Compute" with a "Quality-and-Repetition-aware Information" metric.
3. Parameter Fitting and Validation: The authors fit the quality density and learning decay parameters using small-scale experiments (252M–1.2B) and validate the framework's ability to predict performance on unseen data recipes and larger model scales (up to 7B models trained on 425B tokens).

Results

• Prediction Accuracy: InfoLaw predicts validation loss on unseen data recipes and larger scales (up to 7B models) with a 0.15% mean absolute error and a 0.96% maximum absolute error.
• Extrapolation Capabilities:
  • Unseen Recipes: The model accurately predicts loss for mixture weights not seen during training (e.g., MLQ, MHQ).
  • Larger Scales: Predictions hold for models up to 7B parameters, significantly outperforming traditional scaling laws which diverge at higher compute budgets.
  • Higher Overtraining: The framework successfully extrapolates to a 25x overtraining regime, with the overtraining degree primarily shifting the curve's intercept rather than altering its shape.
• Optimization: Using the fitted InfoLaw, the authors searched for an optimal data recipe for a 2.5B model without additional training runs. The identified recipe outperformed four randomly sampled baselines.
• General Trends: The analysis reveals that smaller models or smaller training budgets prioritize high-quality data, whereas larger models or larger budgets benefit more from data diversity (allocating a smaller fraction to the highest quality bucket).

Significance and Claims

The paper claims that InfoLaw provides a reliable tool for selecting data recipes under varying computational budgets, eliminating the need for extensive trial-and-error experiments. By accurately modeling the interplay between data quality, repetition, and model scale, the framework enables efficient pretraining strategies in data-constrained regimes. The authors assert that their method offers a more robust alternative to traditional scaling laws for predicting performance in scenarios involving weighted-quality mixtures and repetition, facilitating the discovery of optimal data configurations for models ranging from hundreds of millions to billions of parameters.

The paper concludes that while the current data bucketing relies on empirical heuristics and the theoretical explanation for overtraining shifts remains an open question, InfoLaw effectively captures the fundamental trade-offs in LLM scaling, offering a practical path toward more efficient data utilization.