Predictable Emergence: An Empirical Analysis of Whether Sharp Capability Jumps Follow from Smooth Per-Token Scaling Laws

Technical Summary: Predictable Emergence in Multi-Digit Arithmetic

Problem Statement
The paper addresses a central tension in the study of Large Language Models (LLMs): the discrepancy between smooth neural scaling laws and the reported "emergent" capabilities. While pretraining loss follows a predictable power law with respect to model size ($N$), data ($D$), and compute ($C$), practitioners often observe that downstream capabilities (e.g., multi-digit arithmetic, instruction following) appear to jump abruptly from near-zero to high performance at specific scale thresholds. The core question is whether these sharp jumps are genuine, metric-invariant phenomena indicating a new capability, or if they are artifacts of discontinuous evaluation metrics (such as exact-match accuracy) applied to underlying smooth improvements.

Methodology
The authors conduct a narrow, fully controlled, inference-only empirical study to test a falsifiable version of this question: Can the location and shape of an "emergent" jump in exact-match accuracy be quantitatively predicted in advance from the smooth scaling of per-token accuracy?

• Task: Multi-digit integer addition. This task is chosen because it is a canonical example of emergence and, crucially, the answer length ($L$) is an exactly controllable parameter that dictates the metric's nonlinearity.
• Models: The study utilizes public, released checkpoints from three model families without any training or fine-tuning:
  • Pythia: 8 sizes ranging from 70M to 12B parameters (trained on "The Pile").
  • Pythia-deduped: A deduplicated variant of the above.
  • OPT: 5 sizes ranging from 125M to 6.7B parameters (different architecture/training pipeline).
• Metrics:
  • Smooth Metrics: Per-token digit accuracy ($p$), per-token cross-entropy (NLL), Brier score, and normalized edit distance. These are measured under teacher forcing on the correct prefix.
  • Discontinuous Metric: Exact-match accuracy (EM), where the entire output string must match the target.
• Compositional Model: The authors propose a model where exact-match accuracy is the probability of all $L$ tokens being correct.
  • Independent Model: Assumes per-token errors are independent, predicting $EM \approx p^L$.
  • Correlation-Corrected Model: Introduces a single parameter $\rho$ to account for weak positive correlation between token errors (e.g., a model either gets the whole number right or fails multiple digits due to carry chains). This yields an effective length $L_{eff} = 1 + (L-1)(1-\rho)$, predicting $EM \approx p^{L_{eff}}$.
• Forecasting Protocol: The per-token scaling law is fitted using only the four smallest models ($N \le 1B$). This fit is then used to extrapolate and predict the exact-match curves and emergence thresholds for the larger, held-out models.

Key Results

1. Smooth Underlying Scaling: Per-token digit accuracy rises smoothly and monotonically with scale (from 0.47 at 70M to 0.91 at 12B for Pythia), following a clean power law with exponent $\alpha \approx 0.36$. No jumps are observed in continuous metrics (NLL, Brier, edit distance).
2. Quantitative Prediction of Jumps: The compositional model with a single fitted correlation parameter ($\rho = 0.17$) reproduces the entire $6 \times 8$ grid of exact-match curves (across 6 lengths and 8 model sizes) with an $R^2$ of 0.999 and a median absolute error of 0.004.
   • The naive independent model ($\rho=0$) achieves $R^2 = 0.98$ but systematically under-predicts, confirming that weak positive correlation is the primary correction needed.
3. Data Collapse: When plotting exact-match accuracy against the rescaled coordinate $u = L_{eff} \cdot (-\ln p)$, all data points from all lengths, model sizes, and model families collapse onto a single master curve $EM = e^{-u}$ (collapse RMSE $\le 0.013$). This demonstrates that the family of "emergence" curves is a single smooth law reparameterized by length.
4. Successful Forecasting: Fitting the per-token law on models $\le 1B$ successfully forecasts the held-out jumps of larger models. The predicted emergence thresholds match observed thresholds within a mean absolute error of 0.017 in exact-match accuracy and 0.01 dex in threshold location.
5. Falsification Test: The study finds no evidence of "super-compositional" behavior (where exact-match rises faster than $p^{L_{eff}}$ near the threshold) or jumps in continuous metrics. This suggests no genuine, metric-invariant capability switch occurs within the studied scale range.

Significance and Claims
The paper claims to provide a quantitative resolution to the emergence debate for the specific case of multi-digit arithmetic within the 70M–12B parameter range.

• Nature of Emergence: The authors conclude that for this task, "emergence" is not a new capability appearing at a threshold but rather a smooth per-token scaling law viewed through a length-$L$ nonlinearity (the exact-match metric).
• Predictability: The sharp jumps are predictable in advance from cheap, small-scale measurements. The study supplies a recipe: measure per-unit metrics on sub-threshold models, fit the scaling law, estimate the number of sub-decisions ($L$) and a single correlation ($\rho$), and extrapolate.
• Scope: The authors explicitly state they do not claim this resolves the emergence debate for all tasks (e.g., open-ended reasoning or instruction following may not admit such a simple map). However, they argue that for tasks scored by all-or-nothing success over a known number of smoothly improving sub-decisions, the phenomenon is an extrapolation problem, not a guessing game.

The study emphasizes that the "mirage" of emergence is created by the choice of metric, and that continuous, high-resolution metrics reveal the underlying smoothness of capability improvement.