Deriving Neural Scaling Laws from the statistics of natural language

Technical Summary: Deriving Neural Scaling Laws from the Statistics of Natural Language

Problem Statement
While experimental neural scaling laws have empirically guided the development of Large Language Models (LLMs), a quantitative theoretical framework capable of predicting the exponents of these laws for modern LLMs trained on natural language datasets has been lacking. Existing theories, primarily based on kernel methods with fixed feature maps, fail to account for the feature-learning capabilities of modern transformers. Furthermore, previous approaches relying on synthetic data models or toy grammars have not successfully linked the intrinsic statistics of natural language directly to the exponents of loss learning curves at token scales relevant to current LLMs. The central question remains: what statistical properties of language control the power-law exponent relating test loss to the amount of training data?

Methodology
The authors propose a general theoretical framework for the data-limited regime, where model capacity and compute are not the primary constraints, but the amount of training data ($P$) is. The theory decomposes the autoregressive loss into two learning mechanisms: (1) extending the effective prediction time horizon, and (2) improving prediction accuracy within that horizon.

The derivation relies on two key statistical properties of the language dataset:

1. Decay of Next-Token Conditional Entropy ($H_n$): The conditional entropy of the next token given $n$ previous tokens decays as a power law with context length $n$: $H_n - H_\infty \propto n^{-\gamma}$.
2. Decay of Token-Token Correlations: The strength of correlations between tokens separated by a temporal lag $n$ decays as a power law: $\|C(n)\|_{op} \propto n^{-\beta}$.

The theory posits that for a given dataset size $P$, the model can only beneficially leverage tokens up to a data-dependent prediction time horizon $n^*(P)$. This horizon is determined by the signal-to-noise ratio of the token-token correlations; specifically, $n^*(P)$ is the maximum lag where the correlation signal exceeds the sampling noise ($O(P^{-1/2})$). This yields the relationship $n^*(P) \propto P^{1/(2\beta)}$.

Assuming that the model learns to use information within the horizon $n < n^*(P)$ sufficiently fast (the "fast-learning" assumption), the dominant source of excess loss is the finite horizon itself. Consequently, the data-limited scaling exponent $\alpha_D$ for the autoregressive loss is derived as:
$$\alpha_D = \frac{\gamma}{2\beta}$$

The framework also predicts a scaling collapse for individual $n$-gram loss curves $L_n(P)$. When rescaled by the dataset's conditional entropy ($n^\gamma$) and the data threshold ($P/n^{2\beta}$), all $n$-gram learning curves should collapse onto a single master curve.

Experimental Verification
The authors tested this theory on two qualitatively different datasets: TinyStories (synthetic short stories) and WikiText-103 (Wikipedia articles).

1. Estimation of Statistics:

   • $\gamma$ (Entropy Exponent): Estimated by training various architectures (GPT-2 APE, GPT-2 RoPE, LLaMA, Mamba) on increasing dataset sizes and fitting the power-law decay of the $n$-gram loss $L_n$ for small $n$.
     • TinyStories: $\gamma \approx 0.325$
     • WikiText: $\gamma \approx 0.265$
   • $\beta$ (Correlation Exponent): Estimated by measuring the operator norm of the token-token covariance matrix $C(n)$ and fitting its decay.
     • TinyStories: $\beta \approx 0.88$
     • WikiText: $\beta \approx 0.94$

2. Validation of Scaling Laws:

   • Scaling Collapse: The authors plotted $n$-gram losses $L_n$ against the rescaled variables $P/n^{2\beta}$ and $L_n \times n^\gamma$. In both datasets, curves for different $n$ collapsed onto a single master curve, validating the proposed scaling ansatz.
   • Exponent Prediction: Using the measured $\gamma$ and $\beta$, the predicted scaling exponent $\alpha_D = \gamma/(2\beta)$ was compared against the empirical slope of the autoregressive loss $L(P)$ vs. $P$.
     • TinyStories Prediction: $\alpha_D \approx 0.185$. Empirical results matched this slope closely.
     • WikiText Prediction: $\alpha_D \approx 0.141$. Empirical results matched this slope closely.

The agreement held across different model architectures (Transformers with absolute and rotary positional embeddings, LLaMA-style, and Mamba) and varying context lengths, provided the context length did not truncate the data-dependent horizon.

Key Contributions

• First Parameter-Free Theory: The paper provides the first theory that quantitatively predicts data-limited neural scaling exponents for natural language without free parameters or synthetic data models.
• Identification of Determinants: It isolates two intrinsic statistical properties of language (entropy decay $\gamma$ and correlation decay $\beta$) as the sole determinants of the scaling exponent in the data-limited regime.
• Scaling Collapse: It demonstrates that diverse $n$-gram learning curves collapse onto a single master curve when rescaled by these statistical properties, offering a rigorous validation of the learning mechanism.
• Universality Class: The work suggests that deep architectures capable of learning hierarchical structures (like Transformers) belong to a universality class where loss is dominated by the finiteness of the data-dependent prediction horizon, rather than suboptimal learning within that horizon.

Significance and Claims
The authors claim that this work unravels a direct link between the shape of neural scaling laws and the statistical structure of language itself. By showing that the scaling exponent is a function of measurable language statistics ($\alpha_D = \gamma/2\beta$), the paper moves beyond empirical observation to a principled understanding of how LLMs acquire language competence from data.

The paper notes limitations: the theory is tested within finite ranges of dataset sizes and context lengths (up to a few tens of tokens for the effective horizon in their experiments). It acknowledges that shallow networks or kernel methods likely fall outside this universality class due to the curse of dimensionality, and that the "fast-learning" assumption within the horizon may not hold for all architectures. However, for the deep networks tested, the theory provides a precise, first-principles explanation for the observed scaling behavior.