Skaling: Chinchilla's Exponents Meet Kaplan's Coupling

Technical Summary: Skaling – Chinchilla's Exponents Meet Kaplan's Coupling

Problem Statement
Neural scaling laws are foundational for predicting the performance of Large Language Models (LLMs) and optimizing compute budgets. However, the standard formulation, the Chinchilla law (Hoffmann et al., 2022), relies on a structural assumption that model size ($N$) and training data volume ($D$) impact the loss independently. Mathematically, this additive form ($L = A/N^\alpha + B/D^\beta + E$) enforces a zero mixed derivative ($\partial^2 L / \partial N \partial D = 0$).

The authors demonstrate that this independence assumption leads to systematic prediction errors. While the Chinchilla law performs well in the interior of the training grid, it exhibits a "saddle-shaped" residual error at the boundaries (extremes of $N$ and $D$). Specifically, it systematically underestimates loss in some corner cases and overestimates it in others, failing to capture the interaction between model capacity and data volume. This failure is critical for resource allocation, as it leads to inaccurate extrapolations for frontier models where $N$ and $D$ are often imbalanced.

Methodology
To address this, the authors introduce the Skaling law, a generalized functional form that couples model size and data through a single interaction exponent ($k$). The formulation is defined as:

$$L(N, D) = \left( \frac{A}{N^\alpha} + \frac{B}{D^\beta} \right)^k + E$$

• Coupling Mechanism: When $k=1$, the law recovers the purely additive Chinchilla form. When $k \neq 1$, it reintroduces the coupling found in the earlier Kaplan form (Kaplan et al., 2020) but retains Chinchilla's interpretable independent inner exponents ($\alpha, \beta$).
• Theoretical Justification: The authors analyze the loss surface derivatives using Moving Least Squares (MLS) estimators on the Farseer dataset. They show that the mixed derivative $\partial^2 L / \partial N \partial D$ is non-zero and follows a power-law decay, indicating a synergistic effect where scaling $N$ and $D$ together reduces loss more than scaling either alone. The Skaling form naturally reproduces this non-zero mixed derivative.
• Sparse Profiling Strategy (L-shape): The authors propose an efficient sampling strategy to fit these laws. Instead of a dense, square grid, they utilize an "L-shape" grid that restricts training to the low-compute edges: sweeping data volume ($D$) for small models and sweeping model size ($N$) for small data budgets. Theoretical analysis suggests that the asymptotic behavior of the loss function allows these boundary runs to sufficiently constrain the parameters for the entire grid.

Key Contributions

1. Skaling Formulation: Introduction of a single coupling exponent ($k$) that corrects the boundary biases of additive laws while preserving interpretability and the closed-form compute-optimal allocation rule.
2. Sparse Profiling Validation: Demonstration that the Skaling law maintains high predictive accuracy when fitted on an L-shaped grid, requiring approximately 10$\times$ less compute than a full-grid sweep while matching or approaching the accuracy of full-grid Chinchilla fits on interpolation and single-axis extrapolation.
3. Empirical Validation: Rigorous cross-validation across two datasets (Farseer and a custom SK-Grid) showing that Skaling reduces Mean Absolute Percentage Error (MAPE) by 1.5–3$\times$ compared to Chinchilla, particularly in extrapolation regimes.

Results

• Error Reduction: On the Farseer dataset, Skaling reduced the MAPE for single-axis extrapolation from 1.48% (Chinchilla) to 0.47% and for far extrapolation from 2.46% to 2.31%. On the SK-Grid, improvements were even more pronounced, with far-extrapolation error dropping from 5.17% to 0.70% on the full grid and from 14.63% to 1.15% on the L-shape grid.
• Parameter Stability: The fitted coupling exponent $k$ consistently falls in the range of 0.31–0.45 (sub-unit), confirming a non-additive interaction. The irreducible loss floor ($E$) is also systematically lower than in Chinchilla fits, often approaching zero on the Farseer data, suggesting the coupling term absorbs curvature previously attributed to a loss floor.
• Compute Extrapolation: In a test predicting the most expensive runs from cheap data (iso-ratio slices), Skaling achieved a pooled MAPE of 0.60%, outperforming the additive Chinchilla law (2.34%) and the more complex nine-parameter Farseer law (0.80%).
• Allocation Implications: The optimal token-to-parameter ratio ($D^*/N^*$) derived from Skaling differs significantly from Chinchilla's prediction. For the Farseer dataset, Skaling predicts a ratio that decreases with compute scale, whereas Chinchilla predicts a near-flat ratio. At frontier scales, this leads to a 10$\times$ discrepancy in recommended allocation.

Significance
The paper argues that the independence assumption in standard scaling laws is a structural flaw that causes systematic errors exactly where they matter most: the imbalanced extremes of model training. By introducing a minimal coupling parameter, the Skaling law provides a more robust framework for performance prediction.

Crucially, the authors claim that reliable scaling prediction no longer requires massive, dense empirical sweeps. The combination of the Skaling form and the L-shape sampling strategy enables accurate full-grid extrapolation using a fraction of the computational budget. This offers a resource-efficient pathway for allocating compute budgets in next-generation model training, allowing researchers to make confident decisions based on small-scale experiments.