📊 statistics

Quantitative Gaussian-Process limits of Tensor Programs

This paper establishes a quantitative convergence theory for the infinite-width Gaussian-process limits of random neural networks with arbitrary architectures, including weight-sharing schemes, by providing explicit finite-width error bounds of order $O(1/\sqrt{N})$ in Wasserstein distance using the tensor program framework.

Original authors: Andrea Agazzi, Eloy Mosig García, Dario Trevisan

Published 2026-07-08

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Andrea Agazzi, Eloy Mosig García, Dario Trevisan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are baking a giant, complex cake. In the world of artificial intelligence, this "cake" is a neural network, a computer program designed to learn patterns. The "ingredients" are numbers called weights, and the "layers" of the cake are where the magic happens.

Usually, to make a cake, you need a specific, finite amount of flour and sugar. In AI, this is called a finite-width network. It has a set number of neurons (like a set number of mixing bowls) in each layer.

But mathematicians love to ask: "What happens if we make the cake infinitely wide?" What if we had an infinite number of mixing bowls?

The Big Idea: The "Infinite Cake" Limit

The paper by Agazzi, García, and Trevisan is about understanding the relationship between the real, finite cake (the one we can actually build and run on a computer) and the theoretical, infinite cake (a perfect, smooth mathematical object called a Gaussian Process).

For a long time, we knew that as you add more and more layers or make the layers wider, the finite network starts to look more and more like this smooth, infinite mathematical object. It's like how a pixelated image looks blurry and jagged up close, but becomes a smooth, perfect picture when you zoom out far enough.

The Problem: Previous studies told us that they converge, but they didn't tell us how fast or how close they are at any specific size. It was like saying, "Your cake will eventually taste like the perfect one," without telling you if you need 10 extra eggs or 1,000.

The Solution: This paper provides a quantitative recipe. It gives a precise formula for the "error" (the difference in taste) between the finite network and the infinite ideal.

The "Tensor Program" Lens

To solve this, the authors use a tool called Tensor Programs. Think of this as a universal translator.

The Analogy: Imagine you have different types of LEGO sets: a simple house, a complex spaceship, and a robot. They all look different, but they are all built using the same basic rules: snapping blocks together (Matrix Multiplication) and painting them (Non-linear functions).
The Paper's Trick: Instead of analyzing every single LEGO set individually, the authors created a "master language" (Tensor Programs) that describes any network structure—whether it's a simple feed-forward network, a recurrent network (like a memory loop), or even parts of a Transformer (the tech behind modern AI chatbots).
Why it matters: This allows them to prove one big theorem that covers all these different architectures at once, rather than writing a new proof for every new type of network invented.

The Main Result: The "Square Root" Rule

The paper's most important finding is a specific rule about the error.

If you have a network with a width of $n$ (the number of neurons in a layer), the difference between your finite network and the perfect infinite one shrinks at a rate of $1/\sqrt{n}$ .

The Metaphor: Imagine you are trying to guess the average height of people in a city.
- If you ask 4 people, your guess might be way off.
- If you ask 100 people, you are much closer.
- If you ask 10,000 people, you are very close.
- The paper proves that for these neural networks, the "closeness" improves exactly as fast as the square root of the number of neurons increases. If you quadruple the size of your network, you cut the error in half.

Handling the "Tricky" Parts

The paper also tackles two specific complications that make real-world networks messy:

Weight Sharing: In some networks (like those that remember things over time, or "Recurrent Neural Networks"), the same set of weights is reused multiple times, like using the same spoon to stir different bowls. The authors show their math still works perfectly even when the same "spoon" is used over and over.
Attention Mechanisms: Modern AI (like the models that write essays or code) uses "Attention" to focus on specific parts of the input. This involves calculating "kernels" (essentially, how much one part of the data cares about another). The authors extended their math to include these "scalar" variables, proving that even these complex, modern architectures follow the same $1/\sqrt{n}$ rule.

The "Proof" Strategy: Building Line by Line

How did they prove this? They didn't try to look at the whole giant cake at once. Instead, they looked at it line by line.

Imagine the network is a long assembly line.

They start at the beginning (the input).
They prove that if the first step is close to the ideal, the second step will also be close.
They use a technique called coupling. Imagine you have two bakers: one making the real cake (finite) and one making the perfect cake (infinite). The authors show how to make them use the exact same random ingredients (noise) at every step. Because they use the same random noise, any difference in the final cake is purely due to the size of the network, not random luck.

What They Tested (The Experiments)

To make sure their math wasn't just theory, they ran computer simulations. They built networks of different sizes (shallow, deep, recurrent, and residual) and measured how close the output was to the theoretical ideal.

They found that as they made the networks wider, the "distance" between the real output and the perfect output dropped exactly as their math predicted. The graphs showed a clear, straight line on a log-scale, confirming that the $1/\sqrt{n}$ rule holds true even for complex, modern AI structures.

Summary

In short, this paper is a mathematical guarantee. It tells us that no matter how complex your neural network architecture is (as long as it fits within their "Tensor Program" rules), if you make it wider, it will get closer to a perfect, smooth mathematical object. And they tell you exactly how much wider you need to go to get a specific level of accuracy. It turns a vague promise of "bigger is better" into a precise, calculable rule.

Technical Summary: Quantitative Gaussian-Process Limits of Tensor Programs

Problem Statement
The paper addresses the convergence of random neural networks to Gaussian Processes (GPs) in the infinite-width limit. While qualitative Central Limit Theorem (CLT) results for fully connected networks and the Neural Tangent Kernel (NTK) regime are well-established, there is a need for rigorous, quantitative convergence rates for a broader class of architectures. Specifically, the authors aim to provide explicit finite-width error bounds for networks involving weight-sharing schemes (e.g., recurrent networks, transformers) and complex computational graphs, moving beyond simple feed-forward structures. The goal is to quantify the distance between finite-width network executions and their infinite-width Gaussian-process counterparts in the Wasserstein metric.

Methodology
The authors utilize the Tensor Programs formalism, specifically the Netsor language, to abstract neural network architectures. This framework treats networks as sequences of elementary operations (MatMul and NonLin) acting on variables of specific types (H-vars for hidden states, A-vars for weights, and G-vars for pre-activations).

The core methodology involves:

Dual Execution Rules: Defining two modes of execution for a Netsor program:
- Finite-width: Standard matrix-vector multiplication with Gaussian-initialized weights ( $W_{ij} \sim N(0, 1/m)$ ).
- Infinite-width: A limit where matrix multiplications are replaced by their Gaussian central-limit approximations, treating G-vars as jointly Gaussian random variables with covariances defined by recursive kernels.
Quantitative Coupling: The proof employs a line-by-line induction on the program structure. The authors construct an explicit coupling between the finite-width and infinite-width executions.
- For NonLin operations, the error is controlled via the Lipschitz continuity of activation functions.
- For MatMul operations, both executions are conditionally Gaussian given the previous hidden states. The authors couple these conditional laws using the same standard Gaussian noise. The resulting error is reduced to estimating the difference between the conditional means and the square roots of the conditional covariances.
Structural Lemmas: The proof relies on structure lemmas showing that conditional means and covariances are finite-dimensional functions of the Gram matrices of parent variables. The authors utilize the differentiability of these functions at the infinite-width point (after reducing degenerate cases) and apply a quantitative Kernel Law of Large Numbers (LLN) to bound the deviations.
Extension to NetsorK: To cover attention mechanisms and normalization layers, the framework is extended to NetsorK, which introduces scalar kernel variables and scalar-parametric nonlinearities. This requires handling local Lipschitz conditions and ensuring non-degeneracy of the Gaussian limits.

Key Contributions

Quantitative CLT for Tensor Programs: The main result (Theorem 1.2) establishes that for a Netsor program with $L$ lines, the Wasserstein distance ( $W_p$ ) between the scaled finite-width execution and the infinite-width Gaussian process is bounded by $O(\sum_{m \in n_{hidd}} m^{-1/2})$ , where $n_{hidd}$ represents the hidden widths. This provides an explicit inverse square-root convergence rate.
Architecture-Agnostic Framework: The results apply to any architecture expressible as a Netsor program, including feed-forward networks, recurrent neural networks (RNNs) with weight sharing, and networks evaluated on multiple inputs simultaneously.
Kernel LLN: The paper derives a quantitative Law of Large Numbers for empirical kernels (Corollary 1.3), showing that empirical inner products of hidden states converge to their infinite-width expectations with the same $O(n^{-1/2})$ rate.
Extension to Attention: The introduction of the NetsorK language and Theorem 2.4 extends these quantitative guarantees to architectures involving scalar kernel variables (e.g., attention scores) and scalar-parametric nonlinearities (e.g., LayerNorm, Softmax), provided specific non-degeneracy and Lipschitz conditions are met.

Results

Convergence Rates: The theoretical bounds confirm that the error between finite-width networks and their GP limits scales as the inverse square root of the hidden layer widths.
Numerical Validation: Section 5 presents numerical experiments on Multi-Layer Perceptrons (MLPs), Time-unrolled RNNs, and Residual Networks. Using the Sliced Wasserstein-1 distance, the authors observe convergence rates (log-log slopes) that are strictly steeper than $-0.5$ (the sampling noise floor), confirming that the architectural convergence signal is visible. For example, Deep MLPs showed a slope of $-0.916$, and Residual Networks showed $-0.797$.
Degeneracy Handling: The paper demonstrates how to reduce degenerate MatMul lines (where conditional variance vanishes) to NonLin lines, ensuring the validity of the quantitative bounds under non-degeneracy assumptions.

Significance and Claims
The authors position this work as a quantitative counterpart to the qualitative limit theory provided by Yang's Master Theorem for tensor programs. While Yang's theorem offers a broad qualitative description of infinite-width limits, this paper provides explicit, checkable error bounds for a rich class of architectures.

The significance lies in:

Rigorous Finite-Width Analysis: Moving beyond asymptotic statements to provide concrete error estimates for practical, finite-width networks.
Handling Weight Sharing: Addressing the limitations of layer-by-layer independent CLTs by capturing the conditional Gaussian structure essential for recurrent and shared-weight architectures.
Foundation for Future Work: By establishing a quantitative framework for Netsor and NetsorK, the paper lays the groundwork for analyzing more complex modern architectures (like Transformers) and potentially extending these bounds to training dynamics, though the current work focuses strictly on the initialization (forward pass) regime.

The paper remains modest in its claims, noting that the Netsor language does not yet cover all operations in modern architectures without extension, and that the quantitative bounds rely on specific regularity assumptions (Lipschitz continuity) and non-degeneracy conditions.