📊 statistics

Optimal Initialization in Depth: Lyapunov Initialization and Limit Theorems for Deep Leaky ReLU Networks

This paper provides a rigorous probabilistic analysis of deep Leaky ReLU networks to derive a Lyapunov exponent governing activation stability, revealing limitations in standard initialization methods and proposing a novel "Lyapunov initialization" that sets this exponent to zero to ensure optimal training stability.

Original authors: Constantin Kogler, Tassilo Schwarz, Samuel Kittle

Published 2026-06-03✓ Author reviewed ⓘ

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Constantin Kogler, Tassilo Schwarz, Samuel Kittle

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a very tall tower out of blocks. Each layer of the tower represents a "layer" in a neural network (the brain-like computer program). To make the tower stand tall without collapsing or toppling over, you need to start with the right kind of blocks and the right way of stacking them. This paper is about finding the perfect way to stack those blocks so the tower stays stable, no matter how high it gets.

Here is the breakdown of the paper's ideas using simple analogies:

1. The Problem: The Tower is Either Crumbling or Exploding

When you train a neural network, information flows from the bottom (input) to the top (output). The authors discovered that in very deep networks (tall towers), especially those that are narrow (few blocks per layer), the signal traveling through the network tends to do one of two bad things:

Vanishing: The signal gets so weak by the time it reaches the top that it disappears completely. It's like whispering a secret down a line of 100 people; by the time it reaches the end, no one can hear it.
Exploding: The signal gets so loud and chaotic that it blows the tower apart. It's like shouting the secret down the line; the noise becomes so loud it drowns out everything else.

The standard methods people use to start these networks (called "He initialization" or "Orthogonal initialization") are like using a generic recipe for stacking blocks. The paper shows that for narrow, deep towers, this generic recipe often leads to the signal vanishing, making the tower impossible to build.

2. The New Concept: The "Lyapunov Exponent" (The Stability Meter)

The authors introduce a mathematical concept called the Lyapunov exponent. Think of this as a Stability Meter or a Speedometer for the signal.

If the meter reads negative, the signal is shrinking (vanishing).
If the meter reads positive, the signal is growing uncontrollably (exploding).
If the meter reads zero, the signal is perfectly stable. It doesn't shrink or grow; it just flows through the tower at the right size.

The paper proves that for a specific type of activation function (called "Leaky ReLU," which acts like a valve that lets some signal through even when it's small), this meter is the key to understanding what happens as the network gets deeper.

3. The Discovery: Standard Methods Fail in Narrow Towers

The authors did the math to see what the Stability Meter reads when using standard methods.

The Finding: In wide networks (wide towers), the standard methods work fine; the meter reads close to zero.
The Problem: In narrow networks (narrow towers), the standard methods give a negative reading. This means the signal is guaranteed to vanish as the tower gets taller. This explains why training very deep, narrow networks has been so difficult.

4. The Solution: "Lyapunov Initialization"

Instead of guessing, the authors propose a new method called Lyapunov Initialization.

How it works: They calculate the exact settings needed to make the Stability Meter read exactly zero.
The Analogy: Imagine you are tuning a radio. Standard methods tune the radio to a frequency that is slightly off, resulting in static (vanishing signal). Lyapunov Initialization finds the exact frequency where the music is crystal clear. They provide a specific formula to set the weights (the blocks) so that the signal stays stable no matter how many layers you add.

5. The Twist: The "Sampled" Strategy

Even with the meter set to zero, there is a little bit of randomness involved. The paper's math (a "Central Limit Theorem") shows that even in a stable tower, there will be some natural wobbling. The deeper the tower, the more the signal might fluctuate wildly between being too small and too big.

To fix this, they suggest a strategy called Sampled Lyapunov Initialization:

The Analogy: Imagine you are trying to cross a river with stepping stones. Even if you know the path is safe, you might trip on a loose stone. So, instead of trying to cross just once, you prepare many different sets of stepping stones (candidates).
The Action: Before you start training the network, you generate a few different "starter packs" of weights. You test them briefly to see which one keeps the signal closest to the perfect size. You pick the best one and use that to build your tower. This ensures you don't accidentally start with a wobbly foundation.

6. The Results: Building Better Towers

The authors tested their new method on three tasks:

Recognizing handwritten digits (MNIST): Their method helped the network learn much faster and more reliably than standard methods, especially in the early stages.
Learning a complex math formula (Polynomial): Standard methods failed to learn the formula at all (the signal vanished), while their method succeeded.
Learning a "Score" (for AI generation): Their method helped the AI learn the task more efficiently.

Summary

The paper argues that to build very deep, narrow neural networks, we need to stop using generic starting points. Instead, we should use a precise mathematical recipe (Lyapunov Initialization) that guarantees the signal stays stable. If there's still some randomness, we should try a few different starting points and pick the best one (Sampled Lyapunov Initialization). This makes the "tower" of the neural network much more stable and easier to train.

Technical Summary: Optimal Initialization in Depth

Problem Statement

Training deep neural networks requires careful initialization to ensure convergence. While random initialization is standard, existing methods such as Glorot (Xavier) and He initialization rely on assumptions that often fail in deep, low-width regimes. Specifically, these methods aim to preserve the second moment (variance) of activations across layers but do not guarantee the stability of the activation norm itself. In deep networks with low width ( $d$ ) and Leaky ReLU activations, standard initializations often lead to vanishing activations, preventing effective learning. The paper identifies that the growth of activation norms in deep random networks is governed by a parameter known as the Lyapunov exponent, and that standard methods frequently result in a negative exponent, causing exponential decay.

Methodology

The authors provide a rigorous probabilistic analysis of deep, bias-free random neural networks with Leaky ReLU activations ( $\phi(x) = \max(x, \alpha x)$ ). They model the network depth $\ell$ as a stochastic process where the activation at layer $\ell$ is given by $X_\ell = \phi(W_\ell X_{\ell-1})$ , with $W_\ell$ being independent and identically distributed (i.i.d.) weight matrices.

The core of their methodology involves:

Limit Theorems: Instead of analyzing the distribution of $|X_\ell|$ directly, the authors analyze the logarithm of the norm, $\log |X_\ell|$ . They prove a Law of Large Numbers (LLN) and a Central Limit Theorem (CLT) for this quantity.
Lyapunov Exponent Characterization: They establish that as depth $\ell \to \infty$ $ℓ \to \infty$ , $\frac{1}{\ell} \log |X_\ell|$ $\frac{1}{ℓ} lo g ∣ X_{ℓ} ∣$ converges almost surely to a constant $\lambda_{\mu, \phi}$ $λ_{μ, ϕ}$ , the Lyapunov exponent.
- If $\lambda_{\mu, \phi} < 0$ , activations vanish.
- If $\lambda_{\mu, \phi} > 0$ , activations explode.
- If $\lambda_{\mu, \phi} = 0$ , activations are stable in the mean-logarithmic sense.
Explicit Formulas: The authors derive closed-form integral expressions for $\lambda_{\mu, \phi}$ $λ_{μ, ϕ}$ for two common weight distributions:
- Gaussian: Entries sampled from $\mathcal{N}(0, \sigma^2)$ .
- Orthogonal: Matrices sampled from a scaled orthogonal group $\eta \cdot O(d)$ .
Initialization Strategy: Based on these formulas, they propose Lyapunov Initialization, which selects the scaling factor ( $\sigma$ or $\eta$ ) such that $\lambda_{\mu, \phi} = 0$ . They further introduce Sampled Lyapunov Initialization, which generates $O(\sqrt{\ell})$ candidate initializations and selects the one where the expected output norm is closest to 1, mitigating the stochastic fluctuations predicted by the CLT (which scale as $O(\sqrt{\ell})$ ).

Key Contributions

Limit Theorems for Non-Linear Networks: The paper proves a Law of Large Numbers and a Central Limit Theorem for the logarithm of activation norms in deep Leaky ReLU networks. This extends classical results on random matrix products to the non-linear setting, establishing that activation growth is governed by the Lyapunov exponent.
Analytical Formulas: The authors provide explicit, closed-form integral formulas for calculating the Lyapunov exponent for both Gaussian and orthogonal weight matrices.
Critique of Standard Methods: Theoretical analysis reveals that standard He initialization and standard scaled orthogonal initialization yield negative Lyapunov exponents in low-width regimes ( $d$ is small), leading to vanishing activations. Conversely, in the infinite-width limit ( $d \to \infty$ ), these standard methods approach a zero Lyapunov exponent, offering a theoretical justification for their success in high-dimensional settings.
Novel Initialization Schemes:
- Lyapunov Initialization: Sets the Lyapunov exponent to exactly zero to maximize stability.
- Sampled Lyapunov Initialization: A refinement that accounts for depth-dependent stochastic fluctuations by selecting the best candidate from a set of initializations.

Results

The paper presents both theoretical derivations and empirical evidence:

Theoretical: The derived formulas show that for low dimensions (e.g., $d=2$ ) and typical Leaky ReLU slopes (e.g., $\alpha=0.1$ ), He initialization results in a Lyapunov exponent of approximately $-0.82$, indicating rapid vanishing. The proposed critical scaling factors ( $\sigma_{crit}$ and $\eta_{crit}$ ) are calculated to be significantly larger than standard He scaling to counteract this.
Empirical: Experiments on MNIST (100 layers, width 10), polynomial regression (60 layers, width 2), and score learning (30 layers, width 2) demonstrate that the proposed methods outperform standard initialization strategies.
- In the MNIST experiment, Lyapunov methods achieved significantly higher test accuracy (up to 84% for Lyapunov Orthogonal) compared to He initialization (36%) and Glorot-Bengio (12%).
- In polynomial learning, the proposed methods reduced median training loss dramatically compared to baselines, which often failed to learn (stuck near the zero polynomial).
- Sampled Lyapunov methods showed particular advantage in the early training phases and in avoiding the large outliers caused by the CLT fluctuations.

Significance and Claims

The paper claims to provide a rigorous probabilistic foundation for understanding activation stability in deep networks, moving beyond heuristic variance preservation. Its primary significance lies in:

Identifying the Phase Transition: Characterizing the sharp transition between vanishing and exploding activations via the Lyapunov exponent.
Explaining Low-Dimensional Failure: Demonstrating theoretically why standard initialization methods fail in deep, low-width networks (negative Lyapunov exponent) and why they succeed in high-width networks (exponent approaches zero).
Providing a Solution: Offering a theoretically grounded initialization method that explicitly targets the zero-exponent regime, leading to empirically improved learning stability and performance in challenging deep, narrow architectures.

The authors note that their theoretical results are specific to Leaky ReLU (and generalized Leaky ReLU) activations due to the property of positive homogeneity, which is essential for their proof strategy involving spherical stationary measures. They acknowledge that these results do not directly extend to other non-linearities like ReLU (where vanishing can be absolute) or tanh (where the CLT fails).