📊 statistics

Revisiting the Neural Tangent Kernel: the role of large width and depth

This paper challenges the conventional view that the Neural Tangent Kernel (NTK) fails to describe complex overparameterized networks by demonstrating that, contrary to the "lazy training" regime, aggregate neuron activation deviations persist and proper scaling of depth and training time yields non-trivial, generalizable outputs even in infinitely wide and deep networks.

Original authors: William St-Arnaud, Margarida Carvalho, Golnoosh Farnadi

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: William St-Arnaud, Margarida Carvalho, Golnoosh Farnadi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a giant orchestra where every musician is a tiny neuron in a computer brain. For years, scientists believed that if you made this orchestra infinitely large (adding infinite musicians), the music would become boring and static. They thought the musicians would just play their initial notes forever, never changing their tune, no matter how much they practiced. This was called the "lazy regime."

This paper, "Revisiting the Neural Tangent Kernel," challenges that boring conclusion. The authors argue that while individual musicians might barely move, the entire orchestra is actually dancing and evolving in complex ways. They also show that if you add too many layers of musicians (making the orchestra very deep), the music can collapse into a single, repetitive note—unless you know exactly how to conduct it.

Here is a breakdown of their findings using simple analogies:

1. The "Lazy" Orchestra is Actually Dancing (The Width Problem)

The Old Idea: If you have a massive choir (infinite width), each singer stays frozen in their starting position. Because they don't move, they can't learn new songs or patterns. They are "lazy."

The New Discovery: The authors say this view is too narrow. It's like looking at a single ant in a massive colony. The ant might not move much, but the colony as a whole is building a complex castle.

The Analogy: Imagine a stadium full of people holding up colored cards. If you look at one person, their card barely moves. But if you look at the whole crowd, the pattern of colors shifts and changes dramatically to form a picture.
The Result: Even though individual neurons (musicians) stay close to where they started, the combined effect of millions of them creates a dynamic, evolving feature set. The "lazy" label is misleading because the group learns complex patterns even if the individuals seem static.

2. The Tower of Babel Problem (The Depth Problem)

The Old Idea: If you keep stacking layers of neurons (making the network deeper and deeper), the math suggests the network eventually forgets the data entirely. It becomes a "data-independent" predictor.

The Analogy: Imagine playing a game of "Telephone" with 1,000 people. If you pass a message down the line, by the time it reaches the end, the message has been distorted so much that it sounds like random noise or a single, meaningless hum. The network loses its ability to distinguish between different inputs.

The New Discovery: The authors found a way to stop the message from turning into noise. You don't have to let the game run forever.

The Analogy: Think of the network as a runner. If they run forever (infinite time), they get lost. But if you tell them to stop at the perfect moment (early stopping) and adjust the length of the track (depth) based on how many runners are on it (dataset size), they can still deliver a meaningful message.
The Result: By carefully balancing how deep the network is, how much data it sees, and when to stop training, the network remains stable and expressive. It doesn't collapse into a boring, constant answer; it can still distinguish between different inputs.

3. The "Goldilocks" Zone

The paper suggests that the "lazy regime" isn't a dead end. Instead, it's a "Goldilocks" zone:

Not too shallow: It needs enough width to create complex patterns.
Not too deep (without control): If it gets too deep without stopping, it collapses.
Just right: If you scale the depth and stop training at the right time, the network behaves like a well-tuned instrument that can generalize to new data, even if it's theoretically "lazy."

Summary

The authors are telling us: Don't throw out the "lazy" theory just yet.

Individual vs. Group: Just because the individual parts don't change much doesn't mean the whole system isn't learning. The group dynamics are rich and complex.
Depth Control: Deep networks don't have to fail. If you manage the depth and stop training at the right time, they can still be powerful and accurate.

Essentially, they are rewriting the rulebook on how we understand these massive, overpowered computer brains, showing that they are more dynamic and useful than previously thought, provided we understand the math of their size and depth.

Technical Summary: Revisiting the Neural Tangent Kernel

Problem Statement

The Neural Tangent Kernel (NTK) framework describes the learning dynamics of overparameterized neural networks trained with gradient descent, positing that in the limit of infinite width, the network behaves like a kernel method with a fixed kernel. This regime is often termed "lazy training" because individual hidden neuron activations remain arbitrarily close to their initialization values. However, this characterization presents two significant theoretical challenges:

The Width Axis: The "lazy regime" implies that features (hidden neuron activations) are static and random, failing to explain the "feature learning" observed in practical deep networks. Prior work suggests that while the NTK captures complex dependencies, it does so without extracting meaningful features, creating a paradox between its generalization success and its lack of feature evolution.
The Depth Axis: As network depth increases in the infinite-width limit, the NTK converges to a data-independent predictor (a constant matrix), rendering the model unable to distinguish between data points not present in the training set. This suggests a theoretical dead-end for deep networks under standard NTK scaling, where the model loses expressivity and generalization capabilities.

This paper challenges the conclusions that the NTK is purely "lazy" regarding feature learning and that deep NTKs inevitably degenerate into trivial predictors.

Methodology

The authors employ a combination of theoretical asymptotic analysis and empirical validation on fully-connected ReLU networks with standard Gaussian initialization and Mean Squared Error (MSE) loss.

Theoretical Framework

Notation and Definitions: The paper utilizes the recursive definitions of the NTK ( $\Theta^{(L)}_\infty$ ) and the covariance of neurons ( $\Sigma^{(L)}$ ) established by Jacot et al. (2018a) and Arora et al. (2019b). It focuses on the infinite-width limit where the NTK converges to a deterministic kernel.
Feature Analysis (Width): The authors analyze the aggregate behavior of hidden layers rather than individual neurons. They derive a closed-form solution for the inner product of hidden layer activations over time, $\langle h_t(x), h_t(x') \rangle$ , to demonstrate that while individual neuron deviations vanish, the aggregate norm of deviations does not.
Depth Analysis: The paper investigates the limit as depth $L \to \infty$ . It introduces a normalized kernel $\bar{\Theta}^{(L)}_\infty$ and analyzes its convergence properties. The authors derive asymptotic bounds on the eigenvalues of the kernel matrix and the prediction function $f_t(x)$ .
Scaling Strategy: To address the depth degeneracy, the paper proposes a specific scaling regime where the depth $L$ , the stopping time $t$ , and the dataset size $n$ are coupled. Specifically, it avoids the assumption of $t \to \infty$ or $L \to \infty$ in isolation, instead requiring $L$ to scale appropriately with $n$ and $t$ to remain within the "ordered phase" of the kernel dynamics.

Empirical Validation

Toy Models and Datasets: Experiments are conducted on synthetic data, the text8 dataset (using one-hot encodings), and MNIST.
Metrics: The authors measure the deviation of the last hidden layer from its initialization ( $\|h_t - h_0\|_2$ ) and the convergence rates of the kernel values and eigenvalues as depth increases.

Key Contributions

1. Re-evaluating the "Lazy Regime" (Width Axis)

The paper refutes the binary distinction between "lazy" (static features) and "feature learning" (dynamic features) regimes.

Aggregate Dynamics: While individual neuron activations converge to their initialization values as width increases, the aggregate norm of the deviations in the last hidden layer does not vanish.
Theoretical Proof (Proposition 4): The authors prove that for ReLU networks with at least two hidden layers, the inner product of hidden layer activations for distinct inputs, $\langle h_t(x), h_t(x') \rangle$ , is a strictly non-constant function of the training step $t$ . This demonstrates that statistical correlations between data points evolve during training, even in the infinite-width limit.
Implication: The "lazy regime" does not preclude feature learning; rather, the features evolve in a way that is captured by the aggregate layer dynamics, not by individual neuron tracking.

2. Resolving Expressivity in Deep Networks (Depth Axis)

The paper addresses the collapse of the NTK into a data-independent predictor as depth increases.

Convergence to a Constant: The authors confirm that the normalized kernel $\bar{\Theta}^{(L)}_\infty(x, x')$ converges to $1/4$ for $x \neq x'$ as $L \to \infty$ (Theorem 2).
Stabilization via Scaling (Theorem 3): The paper demonstrates that the prediction function $f_t$ $f_{t}$ can remain stable and expressive if the depth $L$ $L$ is scaled relative to the dataset size $n$ $n$ and the stopping time $t$ $t$ is chosen appropriately (early stopping).
- By setting $t \in o(1/\lambda_n)$ (where $\lambda_n$ is the smallest eigenvalue) and scaling $L$ such that it is large enough to stabilize the kernel but not so large that it induces data-independence, the model retains the ability to distinguish data points.
- The authors provide a bound on $L$ dependent on the separation of data points: $L \in O\left(\frac{1}{\sqrt{1-\rho^{(2)}(x_i^\top x_j)}}\right)$ .
Generalization: These results hold for arbitrary datasets on the sphere without assuming the data is generated from a probability distribution with a density function, relaxing constraints found in prior literature (e.g., Xiao et al., 2020).

Results

Feature Evolution: Empirical results on text8 show that while individual neuron deviations are small (consistent with "lazy" behavior), the aggregate deviation of the last hidden layer grows with training, confirming the theoretical prediction of non-static features.
Kernel Convergence: Experiments on synthetic and MNIST data confirm that the kernel values converge to $1/4$ at a rate of $O(1/L)$ .
Stability: By applying the derived scaling laws (Theorem 3), the authors show that the norm of the projection operator in the closed-form solution remains bounded ( $O(n)$ ) and the model maintains expressivity even for large depths, provided the depth is not excessively large relative to the dataset separation.

Significance and Claims

The paper claims to provide a more nuanced theoretical understanding of scaling in neural networks:

Nuance to "Lazy Regime": It argues that the "lazy regime" descriptor is incomplete. While individual neurons are static, the collective layer dynamics allow for feature learning, reconciling the NTK's generalization success with the concept of feature extraction.
Viability of Deep NTKs: It challenges the notion that infinite-depth NTKs are theoretically dead-ends. By properly scaling depth and stopping time, the NTK can yield well-behaved, non-trivial outputs for large datasets.
Practical Alignment: The findings suggest that the "Goldilocks" zone for NTK analysis lies in regimes where depth is large but finite, and early stopping is employed, mirroring practical experimental settings where $L \ll n$ .
Generalization: The authors provide a set of key properties (diagonal dominance, positive definiteness at sufficient depth) that allow these results to be generalized to other kernels and architectures, such as CNNs, though they explicitly note that their analysis does not cover the stochastic fluctuations found in regimes where depth vastly exceeds width (Hanin and Nica, 2020).

The paper concludes that the NTK remains a valuable tool for analyzing overparameterized networks, provided one moves beyond the assumption of static features and infinite training time/depth without scaling.