🔬 condensed matter

Weak Correlations as the Underlying Principle for Linearization of Gradient-Based Learning Systems

This paper proposes that the linearization of gradient-based learning in deep neural networks, particularly in the infinite-width limit, stems from weak correlations between the first and higher-order derivatives of the hypothesis function, a principle used to derive bounds on training deviations and characterized via a novel method for analyzing random tensors.

Original authors: Ori Shem-Ur, Khen Cohen, Aviv Orly, Yaron Oz

Published 2026-08-13

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Ori Shem-Ur, Khen Cohen, Aviv Orly, Yaron Oz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a massive, chaotic orchestra. In the world of artificial intelligence, this orchestra is a "neural network," a computer program designed to learn from data. These networks are made of millions of tiny parts (called parameters) that all talk to each other. Usually, when they learn, they behave like a wild, non-linear storm—changing in complex, unpredictable ways that are incredibly hard to map.

However, scientists have discovered a strange trick: if you make the orchestra big enough (adding so many musicians that it approaches infinity), the chaos suddenly settles down. The network starts behaving like a simple, straight line. This is called the "Neural Tangent Kernel" (NTK) limit. Think of it like a tangled ball of yarn that, when stretched out to a massive size, suddenly looks like a perfectly straight string. For years, researchers have known this happens, but they've been scratching their heads wondering why. Is it magic? Is it just a side effect of the math? And why does this "straight line" behavior sometimes fail to predict how real, finite-sized networks actually perform on tricky tasks?

This paper, titled "Neural Tangent Kernel Perspective on Parameter-Space Symmetries," dives into that mystery. The authors, a team from Tel Aviv University, propose a new way to look at the problem. They suggest that the reason these giant networks turn into straight lines is because of "weak correlations." Imagine the network's parts as a group of friends. In a normal, messy network, everyone is gossiping and influencing each other's decisions. But in these giant, linear networks, the friends barely talk to each other at all. The paper argues that this lack of connection (weak correlation) between the different parts of the network is the fundamental cause of the straight-line behavior. They didn't just guess this; they built a new mathematical toolkit to prove it and showed that when these correlations are weak, the network must behave linearly. They also used computer simulations to show that this theory holds up in practice, helping to explain why some networks learn better than others and why the "infinite" theory sometimes misses the mark in the real world.

The Story of the Quiet Orchestra

To understand what the authors found, let's go back to our orchestra. When a neural network learns, it adjusts its internal knobs (parameters) to get better at a task, like recognizing a cat in a photo. Usually, turning one knob affects many others in a complicated, non-linear way. It's like if the drummer changing their rhythm suddenly made the violinist change their pitch, which then made the bassist switch instruments. It's a mess.

But the authors noticed something special about the "infinite" networks. They realized that for the network to act like a simple, straight line, the "knobs" must stop influencing each other in a deep, tangled way. They call this weak derivative correlations.

Here is a simple way to picture it: Imagine you are trying to predict the weather.

Strong Correlation (The Messy Network): You look at the wind, and it tells you the temperature. But the temperature also tells you the wind, and the humidity tells you both. Everything is connected in a giant, tangled web. Changing one thing ripples through the whole system in a wild, unpredictable curve.
Weak Correlation (The Linear Network): You look at the wind, and it tells you the temperature. But the temperature doesn't really care about the wind anymore. They are mostly independent. If you change the wind, the temperature changes in a simple, predictable, straight-line way.

The paper proves that linearity is exactly the same thing as having these weak correlations. If the parts of the network are "weakly correlated" (they don't talk to each other much), the whole system becomes linear. If they are strongly correlated, it stays messy and non-linear.

The "Chicken and the Egg" Mystery

This discovery solves a big puzzle. For a long time, people thought the network became linear just because it was huge. But this paper suggests it's actually because the specific way these huge networks are built makes their parts independent of each other.

The authors also tackle a tricky question: Is this independence a good thing or a bad thing?

The "Static" View: Some might think that because the parts don't talk to each other, the network is "static" and just memorizes things in a simple way.
The "Bias" View: The authors suggest a deeper idea. They argue that these weak correlations are actually a form of removing bias. If the parts of the network are too connected, they might force the network to learn a specific, weird pattern that isn't actually in the data. By keeping the correlations weak, the network stays "open-minded" and learns the data itself.

However, the paper points out a paradox. While this "static," linear behavior is mathematically beautiful and easy to study, real-world neural networks (which are finite, not infinite) often beat these linear models. Why? The authors suggest that maybe real networks need some connection (some correlation) to learn the tricky, non-linear patterns of the real world. If you force the network to be too linear (too uncorrelated), it might miss out on the "flavor" of the data.

What They Actually Did

The authors didn't just talk about this; they built a new mathematical language to describe it. They introduced a way to measure the "asymptotic behavior" of random tensors (which are just fancy, multi-dimensional grids of numbers that change as the network grows). Think of this as a new ruler that can measure how "wild" or "calm" a network is as it gets bigger.

Using this new ruler, they:

Proved the Connection: They showed mathematically that if the correlations are weak, the network must become linear. It's an "if and only if" relationship.
Checked the Math with Simulations: They ran computer experiments on real datasets (like MNIST and CIFAR10) with different network sizes and learning speeds. They measured how much the real network deviated from the "straight line" prediction.
Found the Limits: They discovered that the "straight line" prediction works great when the network is huge and the learning rate is just right. But if you make the learning rate too fast or the network too small, the "weak correlation" breaks down, and the network goes back to being messy and non-linear.

The Takeaway

The paper suggests that the "magic" of wide neural networks isn't magic at all. It's a result of the parts of the network becoming so numerous that they stop interfering with each other. This "silence" between the parts is what allows the network to behave like a simple, straight line.

But here is the twist: The authors hint that this silence might be why these infinite networks sometimes lose to real, finite ones. Real life is messy and connected. Maybe the best networks aren't the ones that are perfectly silent and linear, but the ones that find a sweet spot—where they are mostly linear but still have just enough "gossip" (correlation) to learn the complex, non-linear secrets of the world.

In short, the paper gives us a new lens to see why big neural networks behave the way they do. It tells us that the key to their simplicity is their lack of connection, but it also warns us that too much simplicity might leave us missing the point.

Technical Summary: Neural Tangent Kernel Perspective on Parameter-Space Symmetries

Problem Statement

Deep learning models, particularly over-parameterized neural networks, are often modeled as non-linear dynamical systems with numerous interacting degrees of freedom. While the Neural Tangent Kernel (NTK) framework established that wide neural networks behave linearly with respect to their parameters in the infinite-width limit, the fundamental mechanisms driving this linearization remain partially understood. Existing literature has debated the "NTK inferiority paradox," where NTK-based learning often underperforms compared to finite-width networks on real-world data. Furthermore, previous theoretical analyses of linearization have often been restricted to specific architectures, relied on averaging correlation values rather than their asymptotic behavior, or failed to provide a unified equivalence between linearization and underlying structural properties. This paper seeks to identify the fundamental cause of linearization in gradient descent-based learning systems and to provide a rigorous, general framework for analyzing deviations from linearity during training.

Methodology

The authors introduce a novel analytical framework centered on weak derivative correlations and a new mathematical tool for characterizing random tensor asymptotic behavior.

Random Tensor Asymptotic Behavior: The paper defines a "Subordinate Tensor Norm" combined with a "Stochastic Big-O Notation" to characterize the asymptotic behavior of random tensors as the system size (limiting parameter $n$ ) approaches infinity. This notation allows for the rigorous bounding of random tensor sequences, addressing challenges related to stochasticity and component complexity. The authors prove the existence and uniqueness of definite asymptotic bounds for these tensors using Zorn's lemma.
Derivative Correlations: The authors define a set of correlation functions, $C_{D,d}(\theta)$ , which measure the correlation between the first derivative of the hypothesis function and its higher-order derivatives (specifically, the interaction between the $D$ -th order derivative and $d$ copies of the first derivative). These correlations are normalized by the learning rate and factorial terms.
Equivalence Theorems: The core theoretical contribution is the establishment of an equivalence between the linearity of the learning dynamics and the weakness of these derivative correlations at initialization.
- Theorem 3.1 & 3.2: The paper proves that for gradient descent-based learning, the system exhibits linear behavior (where the hypothesis function and its derivatives evolve linearly) if and only if the correlations between the first and higher-order derivatives decay at a specific rate (e.g., $O(1/\sqrt{n})$ or $O(1/n)$ ) as the width $n \to \infty$ .
- The proofs utilize Taylor series expansions of the hypothesis function and its derivatives, demonstrating that the evolution of the system is governed by a linear combination of these correlations.
Application to Neural Networks: Using the "Tensor Programs" formalism (extending the work of Yang & Littwin), the authors apply their framework to fully connected neural networks (FCNNs). They demonstrate via induction that wide FCNNs with appropriate initialization and activation functions satisfy the condition of being "weakly correlated," thereby proving they are PGDML (Properly Normalized Gradient Descent-based Machine Learning) systems.

Key Contributions

Fundamental Equivalence: The paper establishes that the linearization observed in wide neural networks is fundamentally equivalent to the existence of weak correlations between the first and higher-order derivatives of the hypothesis function at initialization. This provides a necessary and sufficient condition for linearization, moving beyond sufficient conditions found in prior work.
General Framework: Unlike previous proofs restricted to specific architectures, this approach applies to any system that can be described as a composition of global linear operations and point-wise non-linear functions (including those outside the strict Tensor Programs formalism, such as networks with distinct activation functions per neuron).
Deviation Bounds: The authors derive a bound on the deviation from linearity during Stochastic Gradient Descent (SGD) training over time. This generalizes previous results for deterministic gradient descent, showing that for exponentially weakly correlated systems, the deviation remains bounded by $O(s^0/m(n))$ over $s$ steps.
New Analytical Tool: The introduction of the "Random Tensor Asymptotic Behavior" formalism provides a robust method for analyzing the limiting behavior of random tensors, which the authors argue is applicable beyond deep learning to various mathematical and physical frameworks.

Results

Theoretical Validation: The paper proves that wide neural networks are $\sqrt{n}$ -weakly correlated and power-weakly correlated, confirming that their linear behavior emerges from the structural composition of weak sub-models (neurons becoming independent in the infinite-width limit).
Architecture Sensitivity: The analysis reveals how architectural modifications affect linearization rates. For instance, the $n$ -th derivative of the activation function $\phi^{[n]}$ must satisfy $\phi^{[n]} \le O((n+1)!)$ to ensure the system remains weakly correlated.
Empirical Support: Numerical experiments on MNIST, CIFAR10, and FMNIST datasets validate the theoretical predictions. The experiments compare the training dynamics of non-linear networks against their linearized (NTK) approximations. Results show that the linearization error scales with network width and learning rate scaling, with observed power-law slopes consistent with (though slightly flatter than) theoretical upper bounds.
SGD Dynamics: The derived bounds confirm that while linearization holds, the deviation from the linear limit accumulates over training steps, providing a theoretical explanation for why finite-width networks might diverge from NTK predictions over long training horizons.

Significance and Claims

The paper posits that weak derivative correlations are the foundational cause of the prevalent linearization attributes in wide neural networks. The authors argue that this perspective reframes the understanding of the "NTK inferiority paradox": linear learning may underperform finite neural networks not because linearity is inherently superior, but because finite networks retain beneficial "biases" in the form of non-vanishing derivative correlations that allow them to prioritize realistic data structures over the "overly unbiased" linear limit.

The significance of this work lies in its ability to:

Provide a universal criterion for linearization applicable to generic learning systems, not just neural networks.
Offer a unified explanation for why wide networks linearize, linking it directly to the statistical independence of parameters at initialization.
Enable the analysis of training dynamics (deviation over time) in stochastic settings, which is crucial for practical applications where SGD is the standard optimizer.
Suggest that introducing specific correlations (non-linearities in the bias structure) could be a viable strategy to improve learning in scenarios where strict linearization (NTK limit) is suboptimal.

The authors conclude that their framework offers a deeper, more general understanding of the transition from "feature learning" to the NTK limit, bridging the gap between theoretical infinite-width limits and the empirical success of finite-width deep learning.