📊 statistics

Universality in Deep Neural Networks: An approach via the Lindeberg exchange principle

This paper establishes quantitative bounds on the convergence of fully connected deep neural networks to their infinite-width Gaussian limits by applying a Lindeberg exchange principle to successively replace layer weights with Gaussian random variables.

Original authors: Filippo Giovagnini, Sotirios Kotitsas, Marco Romito

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Filippo Giovagnini, Sotirios Kotitsas, Marco Romito

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the weather. You have a super-complex computer model with millions of tiny sensors (neurons) and connections (weights) all working together. In the real world, these sensors might be a bit "noisy" or imperfect—they might measure temperature with a slight random error, or their sensitivity might vary slightly from one to the next.

This paper is about what happens when you make that computer model huge. Specifically, it asks: If we make the number of sensors in each layer of the network infinitely large, does the messy, noisy model start to behave like a perfectly smooth, predictable mathematical object?

The answer is yes, but the authors wanted to know how fast this happens and how close the messy model is to the perfect one at any given size.

Here is a breakdown of their findings using simple analogies:

1. The "Infinite Crowd" Effect

Think of a deep neural network as a series of relay races.

Layer 1 passes a baton to Layer 2, which passes it to Layer 3, and so on.
In a small network, the baton might get dropped or thrown wildly because the runners (the weights) are unpredictable.
In an infinitely wide network (where each layer has an infinite number of runners), the chaos averages out. The "noise" cancels itself, and the baton follows a perfect, smooth path. Mathematically, this perfect path is called a Gaussian Process (a fancy term for a very predictable, bell-curve-style randomness).

The paper confirms that as you add more runners to each layer, the messy network does converge to this perfect path.

2. The "Lindeberg Switching" Trick

How did they prove this? They used a clever mathematical trick called the Lindeberg Exchange Principle.

Imagine you have a team of 100 runners, and you want to know if their performance is the same as a team of 100 professional athletes who run with perfect, predictable form.

Instead of comparing the whole teams at once, you swap the runners one by one.
You take the first messy runner and replace them with a perfect professional. You check if the team's total time changes much.
Then you swap the second runner, then the third, and so on, until the whole team is made of professionals.

The authors did this mathematically. They started with a network full of "messy" weights (random variables that aren't perfectly Gaussian) and slowly swapped them out for "perfect" Gaussian weights. They calculated the "error" or "distance" introduced at every single swap.

3. The Problem: The "Dimension" Trap

Usually, when you do this swapping trick, the math gets messy very quickly. If you have a huge network, the error tends to explode because there are so many connections. It's like trying to balance a tower of blocks; the more blocks you have, the harder it is to keep it steady.

The authors found that if they just used the standard math, the error would be too big to be useful. The network would have to be impossibly wide to look "perfect."

4. The Solution: The "Smoothing" Secret

The paper's big discovery is that deep neural networks have a built-in smoothing effect.

Without Biases (The Hard Mode): If the network has no "bias" (a constant offset added to every neuron), the math is very strict. To prove the network is close to perfect, the activation function (the rule that decides if a neuron fires) must be incredibly smooth and well-behaved (like a perfectly polished marble). Even then, the network needs to be quite wide to get a good result.
With Biases (The Easy Mode): If the network adds a little bit of "noise" or "bias" at every layer (like adding a tiny bit of static to a radio signal), it actually helps. This extra randomness acts like a lubricant. It smooths out the rough edges of the math.
- The Result: With biases, the authors could prove the network converges to the perfect Gaussian shape much faster, and they didn't need the activation function to be as perfectly smooth.

5. The "Speed Limit" of Convergence

The paper gives a specific formula for how close the messy network is to the perfect one.

They measure the distance using something called 2-Wasserstein distance. Think of this as the "effort" required to move the probability distribution of the messy network to match the perfect one.
They found that the error shrinks as the width of the network increases. Specifically, if you double the width, the error drops by a factor related to the square root of the width.
The Catch: The error depends on the depth of the network (how many layers there are). A deeper network takes a bit longer to "settle" into the perfect shape than a shallow one, but it still gets there.

Summary of the "Takeaway"

The Claim: Randomly initialized deep neural networks behave almost exactly like perfect Gaussian processes when they are wide enough.
The Method: They proved this by mathematically swapping out random weights for perfect Gaussian ones, one layer at a time, and tracking the error.
The Insight: The structure of the network itself helps smooth out the errors, but having "biases" (extra noise) makes this smoothing much more effective, allowing for looser requirements on the network's design.
The Metric: They provided a precise "speed limit" (a mathematical bound) on how fast this convergence happens, showing that the network gets closer to perfection at a rate of roughly $1/\sqrt{\text{width}}$ .

In short, the paper provides a rigorous "receipt" showing that as you build wider and wider neural networks, they inevitably become predictable, Gaussian machines, and it tells you exactly how wide you need to go to get a specific level of predictability.

Technical Summary: Universality in Deep Neural Networks via the Lindeberg Principle

Problem Statement
The paper addresses the theoretical understanding of fully connected deep neural networks (DNNs) in the infinite-width limit. While it is well-established that randomly initialized DNNs converge to Gaussian processes as layer widths tend to infinity (Neal [Nea96]; subsequent works [LSdP+18, dGMHR+18]), a significant gap remains in the quantitative analysis of this convergence for networks with non-Gaussian weight distributions. Previous quantitative results largely focused on Gaussian weights or specific metrics (Total Variation, 1-Wasserstein) under Lipschitz assumptions. The authors aim to establish explicit, quantitative bounds on the convergence rate of the network output to its Gaussian limit in the 2-Wasserstein distance ( $W_2$ ) for general weight distributions, specifically addressing the "universality" phenomenon where the limit holds regardless of the specific weight distribution (provided moments match).

Methodology
The core approach relies on a Lindeberg exchange principle adapted for the recursive structure of deep neural networks. The methodology proceeds through the following logical steps:

Sequential Weight Replacement: The authors define a sequence of intermediate networks where weights are successively replaced layer-by-layer, starting from the output layer and moving backward, with Gaussian random variables that match the first and second moments of the original weights.
Error Accumulation Analysis: The total distance between the original network and the fully Gaussian limit is decomposed into a telescoping sum of differences between consecutive intermediate networks.
Handling Dimensionality: A naive application of the classical Lindeberg principle (Proposition 3.1) yields error bounds dependent on the dimension of the hidden layers (e.g., factors of $n^3$ $n^{3}$ ), which are too large for the DNN setting. The authors overcome this by exploiting the recursive smoothing inherent in the network architecture.
- Zero-Bias Case: They define auxiliary functions $F_{L,K}$ representing the expected value of the test function after conditioning on previous layers. By utilizing the heat semigroup properties and the scaling of Gaussian weights, they demonstrate that the recursive structure provides additional decay factors that counteract the dimension-dependent terms. This requires the activation function $\sigma$ to be highly regular ( $C^{3 \cdot 2^{L-1}}_b$ ).
- Gaussian Bias Case: The presence of Gaussian biases introduces an additional smoothing effect (via convolution with a Gaussian kernel). This allows the authors to bypass the high-order regularity requirements on $\sigma$ , reducing the assumption to $\sigma \in C^3_b$ , while maintaining the same convergence rates.
Metric Conversion: To bound the $W_2$ $W_{2}$ distance, the authors combine the Lindeberg exchange estimates with:
- Classical results on the $W_2$ distance between sums of independent random variables and Gaussians (Bobkov [Bob13]).
- A specific bound on the $W_2$ distance between conditionally independent Gaussian variables with random covariance matrices (Theorem 3.3, derived from [CP25]).

Key Contributions and Results

Quantitative Bounds in $W_2$ : The paper provides explicit upper bounds for the 2-Wasserstein distance between a finite-width DNN output and its infinite-width Gaussian limit.
- Theorem 2.5 (Zero Biases): For a network with zero biases and activation $\sigma \in C^{3 \cdot 2^{L-1}}_b$ , the bound is:
  $W_2(z^{(L+1)}(x), Z^{(L+1)}(x)) \leq C \left( \frac{1}{\sqrt{n_L}} + \left( \sum_{k=1}^{L-1} \frac{1}{\sqrt{n_k}} \right)^{1/4} \right)$
  where $n_k$ are the widths of the hidden layers.
- Theorem 2.6 (Gaussian Biases): For networks with Gaussian biases and $\sigma \in C^3_b$ , the same bound holds. The Gaussianity of biases is crucial here, as it allows for lower regularity assumptions on the activation function.
Universality Principle (Theorem 2.7): The authors establish a weak comparison principle showing that the difference in expectations of test functions between a network with general weights and one with Gaussian weights is bounded by the sum of inverse square roots of the layer widths. This holds for test functions with varying degrees of regularity depending on the presence of biases.
Optimal Rates for Other Metrics: As a corollary (Corollary 2.8), the results imply optimal convergence rates in Total Variation and 1-Wasserstein distances for networks with Gaussian biases, recovering and extending results from [Cel26].
Numerical Validation: Appendix A presents numerical simulations confirming the theoretical decay rates (approximately $O(n^{-1/4})$ to $O(n^{-1/2})$ depending on the metric and configuration) for networks with Laplace-distributed weights.

Significance and Claims
The authors position their work as a refinement of the universality theory for deep neural networks.

Metric Strength: They claim to provide a stronger metric of convergence ( $W_2$ ) compared to previous works focusing on $W_1$ or Total Variation (e.g., [BR25], [Cel26]), which is significant because $W_2$ convergence implies $W_1$ convergence and offers better control over the second moments of the distribution.
Regularity vs. Robustness Trade-off: The paper acknowledges a trade-off: achieving $W_2$ bounds for zero-bias networks requires high regularity ( $C^{3 \cdot 2^{L-1}}$ ) of the activation function, whereas the presence of Gaussian biases relaxes this to $C^3$ . However, the resulting convergence rate of approximately $O(n^{-1/4})$ (dominated by the sum term) is described as "more robust" than the $W_1$ results in [BR25], even if slightly slower than the rates in [Cel26] for specific metrics.
Methodological Novelty: The primary technical contribution is the adaptation of the Lindeberg principle to deep networks. The authors argue that a naive application fails due to dimensionality, and their success relies on exploiting the "additional smoothing" provided by the network's recursive structure and the Gaussian biases.

The paper does not claim to solve the training dynamics of neural networks but strictly focuses on the probabilistic properties of randomly initialized networks in the infinite-width limit, providing a rigorous quantitative foundation for the Gaussian process approximation.