🔢 mathematics

Exploding and vanishing gradients in deep neural networks: the effect of residual connections

This paper utilizes multiplicative ergodic theory and the Furstenberg-Kifer characterization of Liapunov exponents to rigorously analyze how residual connections influence the exploding and vanishing gradient phenomena in deep neural networks.

Original authors: Vivek S Borkar

Published 2026-06-16

📖 4 min read🧠 Deep dive

CC BY 4.0

Original authors: Vivek S Borkar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to pass a message down a very long line of people. In a standard Deep Neural Network (DNN), this is like a game of "telephone" where each person whispers a slightly altered version of the message to the next.

The problem, as this paper explains, is that in a very long line, the message often gets ruined. Sometimes the whisper gets so loud it drowns out everything else (an exploding gradient). Other times, the whisper gets so quiet by the time it reaches the end that no one can hear it at all (a vanishing gradient). This makes it impossible for the network to learn effectively.

This paper, written by Vivek S. Borkar, uses a branch of mathematics called "multiplicative ergodic theory" to explain why this happens and how a specific fix called Residual Connections (used in ResNets) saves the day.

Here is the breakdown of the paper's logic using simple analogies:

1. The Messy Chain Reaction

The author views the layers of a neural network not just as a computer program, but as a dynamical system.

The Analogy: Imagine a ball rolling down a bumpy hill. Each layer of the network is a bump. The "gradient" is the speed and direction of the ball.
The Problem: If the bumps are arranged badly, the ball might accelerate uncontrollably (exploding) or come to a dead stop (vanishing).
The Math Tool: The paper uses something called Lyapunov exponents. Think of these as a "speedometer" that measures the average rate at which the message (or the ball's energy) grows or shrinks as it travels through the network. If the speedometer reads too high or too low, the network fails.

2. The "Identity" Shortcut

The paper focuses on a specific mathematical tool developed by Furstenberg and Kifer. This tool allows the author to look at the "Lyapunov spectrum" (the collection of all these speedometers) and predict exactly how the message will behave.

3. The Magic of the "Residual Connection"

This is the core of the paper. A Residual Connection (ResNet) changes the rules of the game.

Standard Network: The next layer only sees the output of the previous layer.
- Analogy: You are trying to walk forward, but every step you take is completely replaced by a new, potentially shaky step.
Residual Network: The next layer sees the output of the previous layer PLUS the original input. It's a "skip connection."
- Analogy: You are walking forward, but you also have a safety rail (the original input) that you can hold onto. Even if your new step is shaky, you haven't lost your original position.

4. The Paper's Main Discovery

The author proves a specific mathematical fact about this "safety rail."

Without the rail: The "speed" of the message (the Lyapunov exponent) can drift wildly away from 1 (which represents a stable, neutral state). It can shoot up to infinity or crash to zero.
With the rail: The paper proves that adding the residual connection forces the "speed" to stay much closer to the original, stable state (the identity matrix).

The Geometric Metaphor:
The author uses a geometric argument to prove this. Imagine two vectors (arrows) representing the direction of the message.

One arrow is the original direction.
The other is the new, modified direction after passing through a layer.
In a standard network, the new arrow can swing wildly away from the original.
In a ResNet, the new direction is the sum of the original and the modified. Geometrically, this sum creates a new arrow that sits between the original and the modified one.

Because this new arrow is "sandwiched" between the two, it cannot swing as far away as the modified one could on its own. It is mathematically "pulled" closer to the original, stable direction.

The Conclusion

The paper concludes that Residual Connections act as a stabilizer. They don't just "help" the network; they mathematically guarantee that the "Lyapunov exponents" (the growth/decay rates) stay much closer to a safe, neutral value.

In simple terms: Residual connections prevent the message from getting too loud or too quiet by ensuring that the "new" information never completely overwrites the "old" information. This keeps the signal clear all the way to the end of the very long line.

Technical Summary: Exploding and Vanishing Gradients in Deep Neural Networks

Problem Statement
The paper addresses the well-documented phenomenon of exploding and vanishing gradients in Deep Neural Networks (DNNs). During training via stochastic gradient descent (SGD), gradients in certain directions may asymptotically vanish (decay to zero) or explode (grow unbounded in norm). While existing literature offers various explanations and mitigation schemes, this work aims to provide a precise mathematical characterization of this phenomenon and the stabilizing effect of residual connections (ResNets) using multiplicative ergodic theory and Lyapunov exponents.

Methodology
The author, Vivek S. Borkar, reframes the DNN as a discrete-time dynamical system to apply tools from nonlinear dynamics.

Dynamical System Formulation: A DNN consisting of $N$ feedforward blocks is modeled as a composition of functions $f^N_{\beta^N} = f_{\beta_N} \circ \dots \circ f_{\beta_1}$ . The input-output map is treated as the iteration of a dynamical system $x_{n+1} = f_{\beta_n}(x_n)$ .
Linearization: The behavior of gradients is analyzed through the Jacobian matrices of these maps. The product of Jacobians along the network depth corresponds to the evolution of a linear dynamical system driven by random matrices.
Multiplicative Ergodic Theory: The core analytical tool is the Furstenberg-Kifer theorem [8]. Unlike previous works that apply multiplicative ergodic theory more broadly, this paper utilizes the specific exact characterization of Lyapunov exponents provided by Furstenberg and Kifer. This involves:
- Defining the problem on the projective space $P^{d-1}$ .
- Analyzing the set of stationary measures ( $\mu$ -stationary measures) for a Markov chain induced by the random matrices on the projective space.
- Characterizing the Lyapunov spectrum ( $\gamma_i(\mu)$ ) as the discrete values taken by the integral $\alpha(\mu, \nu) = \int \int \log(\|Au\|) \mu(dA)\nu(du)$ over these stationary measures.

Key Contributions and Results
The primary contribution is the application of the Furstenberg-Kifer characterization to derive a precise statement regarding the Lyapunov spectrum of DNNs and the specific impact of residual connections.

Theoretical Framework: The paper establishes that the Lyapunov spectrum of a DNN can be described via a direct sum decomposition of the state space and a sequence of invariant subspaces, determined by the stationary measures of an associated Markov chain on the projective space.
Analysis of Residual Connections: The paper models a residual connection as a transformation where the map $x \mapsto f_{\beta}(x)$ is replaced by $x \mapsto x + f_{\beta}(x)$ .
Main Theorem (Theorem 4.1): The paper proves that the Lyapunov spectrum under a residual connection is a "smaller perturbation" of the spectrum of the identity matrix (which corresponds to a spectrum of all 1s) compared to a network without residual connections.
- Mechanism: The proof utilizes geometric arguments in the projective space. It demonstrates that for a vector $\hat{u}$ , the mapping $(I+A)\hat{u}$ results in a direction that is geometrically closer to the original direction $\hat{u}$ than the direction resulting from $A\hat{u}$ alone.
- Implication: This geometric proximity implies that the integral of the logarithmic growth rate (the Lyapunov exponent) is reduced in magnitude when residual connections are present. Specifically, the difference between the expected log-norm growth with and without the identity term is negative, indicating a stabilizing effect.

Significance and Claims
The paper claims to offer a "different take" on the exploding/vanishing gradient problem by moving beyond qualitative explanations to a "precise mathematical fashion" using projective geometry and the Furstenberg-Kifer theorem.

Precision: The work distinguishes itself from prior studies (such as [13], [6], and [3]) by using the exact characterization of the Lyapunov spectrum to "pin down" the effect of residual connections in greater detail.
Quantitative Stabilization: The result provides a quantifiable explanation for the empirical observation that ResNets are easier to train: residual connections mathematically constrain the Lyapunov exponents to remain closer to zero (or the identity spectrum), thereby preventing the extreme growth or decay of gradients.
Scope and Limitations: The author notes that the analysis relies on specific assumptions, including the existence of a limit state $x_\infty$ (Assumption 2) and integrability conditions on the matrix norms (Assumption 1). The paper acknowledges that while the model is stylized (e.g., assuming smooth activations rather than ReLU and constant dimensions), the mechanism demonstrated is plausible. The authors state that extending these results to full generality (e.g., non-smooth activations, varying dimensions) requires further sophisticated mathematics and is a task for future work.

In summary, the paper argues that residual connections function by geometrically constraining the evolution of gradient directions in the projective space, thereby keeping the Lyapunov spectrum of the network's Jacobian product closer to that of the identity matrix, which mathematically explains the mitigation of exploding and vanishing gradients.