🤖 machine learning

Criticality and Saturation in Orthogonal Neural Networks

This paper provides a theoretical explanation for the stability of finite-width orthogonal neural networks by deriving explicit layer-wise recursion relations and extending Feynman diagram techniques to show that their finite-width tensors stabilize at large depths, a finding validated by excellent agreement between analytical predictions and Monte-Carlo simulations.

Original authors: Max Guillen, Jan E. Gerken

Published 2026-05-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Max Guillen, Jan E. Gerken

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a skyscraper, but instead of steel beams, you are building it out of layers of neurons. In the world of Artificial Intelligence, these "neurons" are connected by weights (numbers) that determine how information flows from the bottom to the top.

For a long time, engineers building these digital skyscrapers had a rule of thumb: when they started construction, they should pick the numbers for the connections completely at random, like rolling dice. This usually worked okay, but sometimes the building would wobble so much that it collapsed (the "exploding gradient" problem) or became so stiff it wouldn't move at all (the "vanishing gradient" problem).

Recently, builders discovered a secret trick: instead of rolling dice, they should pick numbers that are orthogonal. In math-speak, this means the connections are perfectly balanced, like a set of arrows pointing in directions that are perfectly perpendicular to each other. When they did this, the buildings became incredibly stable and trained much faster.

The Problem:
While everyone knew this trick worked in practice, no one could explain why it worked, especially for buildings that weren't infinitely wide. Previous theories only worked for "infinite" buildings or simple, straight-line structures. Real-world buildings are finite (they have a specific width) and curved (they use non-linear activation functions like tanh). The gap in knowledge was: Why does this orthogonal trick keep finite, wobbly buildings stable?

The Solution (The Paper's Contribution):
The authors of this paper acted like master architects and physicists. They built a new set of blueprints using a method called Feynman diagrams. You might know these from particle physics, where they draw little squiggly lines to track how particles bump into each other. Here, the authors used these diagrams to track how information bumps through the layers of a neural network.

They created a new "grammar" for these diagrams that specifically accounts for the "orthogonal" nature of the weights. Think of it like adding a special rule to a game of chess: "If you move a piece orthogonally, the board reacts differently."

What They Found:

The Stability Mechanism: They proved mathematically that when you use these orthogonal weights, the "statistical noise" (the wobble) in the network doesn't grow out of control as the building gets taller. Instead, it hits a "ceiling" and stabilizes.
The "Saturation" Effect: They showed that in very deep networks, these orthogonal structures reach a state of "saturation." Imagine a sponge soaking up water; eventually, it can't hold any more. Similarly, the network's internal statistics stop changing wildly and settle into a predictable, stable pattern. This happens even when the network is wide but not infinite.
The Critical Point: They identified a specific "sweet spot" (called criticality) where the network is perfectly balanced. If you are at this spot, the network is stable. If you are off the spot, it becomes unstable. They showed that the orthogonal method makes it much easier to stay at this sweet spot compared to the random "dice-rolling" method.

How They Proved It:
They didn't just do math on paper. They built a computer simulation (a "Monte Carlo" experiment) where they built thousands of these digital skyscrapers. They measured the wobble layer by layer.

The Result: The computer measurements matched their new mathematical blueprints perfectly. The "orthogonal" buildings stayed stable and behaved exactly as their new theory predicted, while the "random" buildings behaved chaotically.

In a Nutshell:
This paper provides the missing "instruction manual" for why using orthogonal weights makes deep neural networks stable. It bridges the gap between the theoretical world of infinite networks and the practical world of finite, real-world networks. It confirms that this "orthogonal trick" isn't just a lucky guess; it's a fundamental property of how information flows through balanced, finite structures, ensuring they don't collapse or freeze as they get deeper.

Technical Summary: Criticality and Saturation in Orthogonal Neural Networks

Problem Statement

It is well-established that initializing neural network weight matrices to be orthogonal (satisfying $W W^\top = I$ ) rather than using independent and identically distributed (i.i.d.) Gaussian components can improve training performance. However, the theoretical understanding of this phenomenon has been limited. Previous studies were largely restricted to linear networks or relied on mean-field theory in the infinite-width limit. Crucially, at infinite width, the Neural Tangent Kernel (NTK) for networks with orthogonal and Gaussian initializations is identical, rendering infinite-width analysis insufficient to explain the observed differences in finite-width, nonlinear networks.

While recent empirical work (Day et al.) demonstrated that tensors governing finite-width statistics stabilize for large depths in orthogonal networks (unlike in Gaussian networks), a rigorous theoretical derivation explaining this stability was missing. Specifically, no analysis had successfully shown that orthogonal initializations of nonlinear, finite-width networks lead to improved performance or stability over Gaussian initializations through a systematic expansion.

Methodology

The authors employ a finite-width correction framework, analyzing network statistics as a power series in $1/n$ , where $n$ is the network width. The zero-order term corresponds to the infinite-width limit, while higher-order terms capture finite-width effects.

Key methodological components include:

Orthogonal Weingarten Calculus: Instead of Wick's theorem (used for Gaussian weights), the authors utilize orthogonal Weingarten functions to characterize the joint moments of weight matrices sampled from the orthogonal group with respect to the Haar measure. This introduces non-vanishing higher-order cumulants for the weights, which are central to the finite-width statistics.
Layer-wise Recursion Relations: The authors derive explicit recursion relations for the tensors governing the first-order corrections ( $O(1/n)$ ) to the training dynamics. These tensors include the NTK mean correction ( $\Theta^{(1)}$ ), the NTK-preactivation cross-correlators ( $D, F$ ), and the NTK variance tensors ( $A, B$ ). They also derive the recursion for the six-point preactivation cumulant tensor ( $V_6$ ) at order $O(1/n^2)$ .
Feynman Diagrammatic Framework: To manage the algebraic complexity of deriving these recursions, the authors extend a recently introduced Feynman diagram framework (originally for Gaussian weights) to the orthogonal case. This extension introduces:
- Orthogonality Charge: A label tracking the orthogonal character of lines.
- Square Propagators: Representing the full expectation value over weights.
- Selection Rules: Specific rules (e.g., color conservation) that determine which diagrams contribute at a given order.
- Weingarten Weights: Diagrams are weighted by Weingarten functions determined by the relative ordering of pairings.
  The authors prove that this diagrammatic formalism is complete to all orders in $1/n$ .

Key Contributions

Derivation of Recursion Relations: The paper provides the first explicit layer-wise recursion relations for the tensors $D, F, A, B, P, Q, R, S, T, U$ and the first correction to the NTK mean ( $\Theta^{(1)}$ ) at order $O(1/n)$ for orthogonal initializations. It also derives the recursion for the $V_6$ tensor at order $O(1/n^2)$ .
Extension of Feynman Rules: The authors provide a set of Feynman rules specifically for orthogonal weights that simplify the computation of these recursion relations. They prove the completeness of these rules to all orders in $1/n$ and demonstrate their utility by computing the $V_6$ correction, which would be algebraically prohibitive via direct methods.
Theoretical Explanation of Stability: By solving the derived recursion relations (both numerically for single-input cases and analytically via large-depth expansions), the authors show that the theory reproduces the empirically observed stability and saturation of finite-width tensors in orthogonal networks.
Criticality Analysis: The paper extends criticality results for the Neural Network Gaussian Process (NNGP) and NTK from Gaussian to orthogonal initializations, demonstrating that criticality of the infinite-width limit implies criticality of the orthogonal preactivation and NTK cumulants.

Results

Stability and Saturation: Numerical solutions of the recursion relations for tanh-MLPs (width $n=50$ , depth $L=10$ ) show excellent agreement with Monte Carlo estimates from network ensembles. The results confirm that normalized NTK tensors ( $D, F, A, B$ ) and preactivation cumulants attain smaller magnitudes than their Gaussian counterparts and exhibit early-layer saturation.
Large-Depth Behavior: Analytical large-depth expansions ( $\ell \to \infty$ ) of the solutions align with the exact numerical solutions and Monte Carlo simulations, particularly in the regime $\ell > n$ . This confirms that orthogonal networks exhibit saturation at large depths, a behavior not seen in Gaussian networks.
Criticality: The study verifies that at the critical point ( $C_W = 1$ ), the NNGP and NTK remain stable across varying depths, consistent with the bootstrap structure where lower-rank tensor criticality controls higher-rank tensors.
Diagrammatic Validation: The Feynman diagram approach successfully reproduces known algebraic results for Gaussian weights (by restricting to diagonal Weingarten contributions) and correctly generates the additional terms required for orthogonal weights.

Significance

This work provides the first theoretical explanation for the stability of nonlinear, finite-width neural networks initialized with orthogonal weights, closing a long-standing gap in the literature. By bridging the gap between empirical observations (Day et al.) and theoretical analysis, the paper demonstrates that the stability of orthogonal networks is a direct consequence of the specific structure of their finite-width corrections, governed by orthogonal Weingarten functions. The development of a complete Feynman diagrammatic framework for orthogonal weights offers a powerful tool for systematically computing network statistics to arbitrary orders in $1/n$ , facilitating future theoretical investigations into deep learning dynamics.