Technical Summary: Finite-Size Gradient Transport in Large Language Model Pretraining

Problem Statement

While neural scaling laws have established robust power-law dependencies between Large Language Model (LLM) pretraining loss and external factors (model scale, dataset size, compute), they do not reveal the internal organization of the gradient field as training progresses. Scalar performance curves can mask internal reorganizations that unfold over time and across scales. Existing studies on "grokking" (delayed generalization) and critical signal propagation suggest that behavioral transitions may be preceded by smoother internal changes, yet current metrics lack a scale-resolved probe to measure how gradient-field organization evolves. The authors argue that a single exponent (such as the cascade-size exponent $D$) is insufficient to characterize real LLM training regimes, necessitating a multi-channel framework to distinguish between different training dynamics that may share similar size-scaling backbones.

Methodology

The paper introduces a finite-size gradient-transport framework grounded in non-equilibrium statistical mechanics, specifically adapting the Thresholded Diffusion Update–Olami-Feder-Christensen (TDU-OFC) avalanche probe.

1. The TDU-OFC Probe

The method treats the gradient field (or parameter update field) as a load field subjected to a controlled redistribution rule on a fixed Barabási–Albert (BA) graph.

• Input: At each training step $t$, a field snapshot $u(t)$ is taken. For Pico-LM, this is the raw gradient vector $g_t$; for Pythia, it is the checkpoint-difference update field $\Delta\theta_t$.
• Thresholding: A fixed threshold $\tau(t)$ is set at the 90th percentile of the field magnitude.
• Relaxation: Active nodes (where $|u_i| > \tau$) relax synchronously, redistributing signed values to neighbors based on degree normalization. This process iterates until no active nodes remain or a step ceiling (500) is reached.
• Outputs: The probe yields the cascade size $s(t)$ (total active events) and cascade duration $n_{steps}(t)$.

2. Five-Quantity Characterization

The framework extends the single exponent $D(t)$ to a five-quantity algebraic closure:

1. $D(t)$ (Cascade-size exponent): Describes spatial extensivity ($s_{max} \sim N^D$).
2. $z(t)$ (Duration exponent): Describes temporal persistence ($n_{steps} \sim N^z$).
3. $\beta$ (Absolute transport exponent): Derived from $D - z$.
4. $\delta$ (Intensive transport exponent): Describes per-parameter efficiency ($v_{rel} \sim N^\delta$).
5. $v_{rel}$ (Relative transport efficiency): A directly measured observable defined as $s_{max} / (N \cdot n_{steps})$.

These quantities satisfy the algebraic closure relations: $\beta = D - z$ and $\delta = \beta - 1 = D - z - 1$.

3. Empirical Families and Null Controls

The framework is applied to two complementary families:

• Pico-LM: Four scales (11M–570M) with direct raw-gradient snapshots at 125 aligned steps.
• Pythia/PolyPythias: Five sub-1B scales (14M–410M) using checkpoint-difference fields (proxy for updates) at 153 aligned steps.

To ensure comparability, the authors employ a null decomposition hierarchy:

• $N_0/N_1$: Gaussian nulls (standard or moment-matched) to establish a baseline.
• $N_2$: A permutation null preserving the signed marginal distribution but randomizing value-to-node assignment.
• This allows the separation of "distribution effects" from "assignment effects" (the specific organization of values on the graph).

4. Stepwise Compressibility

A novel contribution is treating stepwise compressibility (the $R^2$ of the power-law fit at each step) as an observable of internal organization, rather than a mere fit-quality footnote. This measures whether a transport channel admits a clean, single cross-scale power-law slope at a given training step.

Key Results

1. Shared Size Backbone, Distinct Transport Regimes

Both Pico-LM and Pythia exhibit a near-unity cascade-size backbone ($D \approx 1$) throughout the stable training window. However, they occupy distinct transport regimes:

• Pico-LM: Shows positive duration scaling ($z > 0$) and negative intensive-efficiency scaling ($\delta < 0$). Smaller models exhibit higher per-parameter transport efficiency, and cascades last longer in larger models.
• Pythia: Remains in a narrow band near the $D=1$ baseline with weak positive efficiency scaling ($\delta > 0$) and near-zero duration scaling ($z \approx 0$). It approaches a common-efficiency band across scales.

2. Null Baseline Alignment

Randomized field controls ($N_0$) reveal that both families share nearly identical null floors in the intensive ($v_{rel}$) and duration ($z$) channels. The observed contrasts are therefore interpreted as different real departures from a shared null skeleton rather than artifacts of different null calibrations. In the duration channel, Pico shows a positive departure from the null, while Pythia shows a negative departure.

3. Compressibility as an Observable

• Pico-LM: Exhibits high stepwise compressibility ($R^2 \approx 0.97-0.99$) across size, duration, and efficiency channels, indicating clean power-law organization at every step.
• Pythia: While the size backbone ($D$) remains highly compressible ($R^2 \approx 1.0$), the duration and efficiency channels show weak one-slope compressibility ($R^2 \approx 0.55-0.65$). This suggests that Pythia's internal transport channels are not well-described by a single power-law slope at each step, even though the aggregate behavior is stable.

4. Association with External Performance

The paper finds channel-level associations but no significant exponent-level associations:

• Channel Level: External performance (perplexity for Pico, zero-shot accuracy for Pythia) is strongly associated with the intensive efficiency observable ($v_{rel}$) and normalized duration ($n_{steps}/N$). Better performance correlates with lower $v_{rel}$ and higher normalized duration. These associations persist after controlling for the learning rate schedule.
• Exponent Level: The size exponent $D(t)$ acts as a shared backbone but does not correlate significantly with external performance exponents ($\beta_{PPL}$ or $\alpha_{acc}$) across the full trajectory in either family.

Significance and Claims

The paper claims to provide a reusable transport measurement framework for comparing real LLM training regimes without asserting a universal fixed point or a first-principles derivation of neural scaling laws.

• Shift in Role for $D(t)$: In controlled "grokking" benchmarks (previous work), $D(t)$ acted as a transition-localized marker (crossing $D=1$ at generalization). In real LLM pretraining, $D(t)$ becomes a stable backbone coordinate ($D \approx 1$) that does not distinguish regimes; the distinction lies in how this backbone is decomposed into duration and efficiency channels.
• Observational Nature: The results are observational and finite-scale. The authors explicitly state they do not claim that LLM training realizes a classical Self-Organized Criticality (SOC) mechanism or a full crackling-noise universality class.
• Methodological Contribution: The work elevates "stepwise compressibility" to the status of a primary observable, arguing that the degree to which internal channels admit clean power laws is a fundamental property of the training dynamics' organization.
• Limitations: The authors caution that the Pico/Pythia contrast involves multiple variables (raw gradient vs. update field, different architectures, optimizers). Strong quantitative claims are restricted to within-family analyses, while cross-family comparisons are limited to regime-level structures.

In summary, the paper argues that real pretraining is better described by a multi-observable transport structure (size, duration, efficiency, and compressibility) than by any single crossing statistic, offering a physics-inspired language to resolve internal reorganizations that scalar loss curves hide.