Inverse Depth Scaling From Most Layers Being Similar

Technical Summary: Inverse Depth Scaling From Most Layers Being Similar

Problem Statement
Neural scaling laws have established that large language model (LLM) performance improves predictably with increases in model size and dataset size, typically following power-law relationships. However, these laws often treat model size as a monolithic quantity, obscuring the distinct contributions of width and depth. While theoretical frameworks propose that depth could contribute to performance through compositional assembly (hierarchical abstraction), procedural assembly (discretization of smooth dynamics), or ensemble averaging (redundant estimation), the actual mechanism by which LLMs utilize their depth remains unclear. Empirical studies suggest LLM layers are often redundant, yet a quantitative framework linking this redundancy to specific scaling laws has been lacking. This paper investigates the quantitative relationship between LLM depth and loss to determine which theoretical regime dominates.

Methodology
The authors employ a three-pronged approach combining empirical analysis of real LLMs, controlled toy model experiments, and theoretical analysis:

1. LLM Hidden State Analysis: The authors analyzed hidden state dynamics across layers in various Pythia models (ranging from 410M to 12B parameters) and other families (Qwen-2.5, OPT) using the FineWeb dataset. They measured the angular change between neighboring hidden states, $\theta(h_l, h_{l+1})$, and the correlation between incremental updates, $\theta(\Delta h_l, \Delta h_{l+1})$. Principal Component Analysis (PCA) was used to identify distinct behavioral clusters among tokens.
2. Empirical Loss Decomposition: The authors proposed a decomposed scaling law for loss ($L$) separating contributions from width ($m$), depth ($\ell$), and dataset size ($D$):
   $$L = \frac{c_m}{m^{\alpha_m}} + \frac{c_\ell}{\ell^{\alpha_\ell}} + \frac{c_D}{D^{\alpha_D}} + L_0$$
   They fitted this model to data reconstructed from Chinchilla scaling experiments (approx. 200 data points) to estimate the depth scaling exponent $\alpha_\ell$.
3. Toy Model Experiments: To isolate mechanisms, a teacher-student setup was designed where a student network learns to mimic a deeper teacher. The teacher's dynamics were controlled via weight tying (inducing smooth dynamics for procedural assembly) or independent weights (inducing non-smooth, random-walk dynamics for ensemble averaging). The student's depth was varied, and training was conducted under different target distribution temperatures to observe convergence and scaling behaviors.

Key Contributions and Results

• Inverse Depth Scaling: The primary finding is that LLM loss scales approximately inversely proportional to depth ($\alpha_\ell \approx 1$). The fitted exponent for the depth-dependent term is $\alpha_\ell = 1.2 \pm 0.3$, suggesting a power-law decay where doubling the depth roughly halves the depth-limited error component.
• Dominance of Ensemble Averaging: Analysis of hidden states reveals that for the vast majority of tokens (99.6%) and layers, updates are incremental and uniform across the middle of the network. Crucially, the correlation between neighboring layer updates is weak (angles close to $\pi/2$), which contradicts the smooth dynamics required for procedural assembly. Instead, these signatures align with ensemble averaging, where layers act as redundant, noisy estimators that reduce error through averaging.
• Rejection of Compositional and Procedural Dominance:
  • Compositional Assembly: The data does not show the "early stopping" or distinct hierarchical abstraction patterns (e.g., syntax $\to$ semantics $\to$ reasoning) predicted by this regime for the majority of data.
  • Procedural Assembly: While procedural assembly (discretization of smooth ODEs) can theoretically yield inverse scaling under specific conditions (e.g., low temperature/incomplete training), the authors' toy model experiments show that once training converges with smooth dynamics, the scaling exponent shifts to $\alpha_\ell \approx 3$. Since real LLMs exhibit $\alpha_\ell \approx 1$, procedural assembly is unlikely to be the dominant mechanism in converged LLMs.
• Mechanistic Explanation: The paper concludes that the inefficiency of LLM depth arises because most layers perform functionally similar transformations. Adding depth increases the size of these functional groups, and the loss reduction is driven by the law of large numbers (averaging out errors) rather than the construction of new, deeper abstractions.

Significance and Claims
The authors claim this work provides a quantitative explanation for the "inefficient" use of depth observed in prior empirical studies. By establishing that LLMs operate primarily in an ensemble averaging regime, the paper suggests that:

1. Architectural Implications: Current residual network architectures may inherently bias models toward ensemble averaging rather than compositional learning. Improving efficiency may require architectural innovations that encourage the compositional use of depth.
2. Scaling Laws: The observed inverse depth scaling ($\alpha_\ell \approx 1$) combined with width scaling ($\alpha_m \approx 1$) implies an optimal width-depth relationship of $m \propto \ell$ under a fixed parameter budget. This aligns with the empirical $N^{-1/3}$ scaling observed in Chinchilla models.
3. Limitations: The authors modestly note that their loss decomposition was proposed based on empirical fits rather than derived from first principles. They acknowledge that while they rule out the dominant role of compositional and procedural assembly, other mechanisms producing similar scaling behaviors cannot be rigorously excluded without first-principle derivations. Additionally, the study focuses on the "bulk" of layers, acknowledging that the first and last layers may exhibit different, potentially compositional, behaviors.

In summary, the paper posits that the current success of scaling LLMs is largely a result of adding redundant layers that average out errors, rather than deepening the representational hierarchy, and that future architectural progress may depend on breaking this bias toward ensemble averaging.