📊 statistics

Explicit integral representations and quantitative bounds for two-layer ReLU networks

This paper presents a method for constructing explicit integral representations for two-layer ReLU networks that can approximate multivariate polynomials with $L^2$ error bounds that depend on monomial coefficients and the distribution rather than explicitly on dimension or degree.

Original authors: Anthony Lee

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anthony Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to explain a complex, beautiful landscape to a friend, but you are only allowed to use one specific tool: a LEGO brick.

You can’t use clay, you can’t use paint, and you can’t use digital pixels. You only have these rigid, rectangular blocks. If you want to recreate the smooth curves of a rolling hill or the sharp peak of a mountain, you have to stack thousands of these little bricks in just the right way. If you use enough of them, and you arrange them carefully, the "staircase" effect disappears, and the landscape looks smooth from a distance.

This paper is essentially a mathematical "instruction manual" for how to build any complex shape (specifically, mathematical "polynomials") using only the "LEGO bricks" of the AI world: ReLU neurons.

Here is the breakdown of the paper’s big ideas:

1. The "LEGO Brick" (The ReLU Neuron)

In Artificial Intelligence, a "neuron" is a tiny mathematical function. The most common type is called ReLU. Think of a ReLU neuron as a dimmer switch that only works in one direction. It stays at zero until you hit a certain level, and then it turns on and goes up in a straight line.

By itself, a ReLU neuron is very boring—it’s just a straight ramp. But if you combine millions of these ramps, you can approximate almost anything.

2. The "Master Blueprint" (Integral Representations)

The author, Anthony Lee, asks a profound question: Is there a perfect recipe that tells us exactly how to stack these ramps to recreate a specific shape?

Usually, scientists try to figure this out by looking at "frequencies" (like how music is made of different notes). But Lee takes a different route. He treats the network like an integral—which is basically a way of saying "a massive, continuous sum."

He discovers a "Master Blueprint" (an Integral Representation). Instead of guessing where to put the bricks, his math provides a formula that says: "To build this specific mountain, you need this many bricks at this specific angle, with this much weight."

3. "Sharpening" the Bricks (The Heat Equation)

One of the coolest parts of the paper is a trick called "Sharpening."

Imagine your LEGO bricks are a bit chunky and imprecise. To make the mountain look smoother, you could "heat up" the bricks so they melt slightly and settle into the gaps, making the surface more refined.

Lee uses something called the Heat Equation (the math that describes how warmth spreads through a metal rod) to "denoise" the function. By mathematically "running the heat backward," he can transform a clunky, jagged representation into a "sharpened" one that is much more efficient and accurate. It’s like taking a blurry photo and using AI to make it high-definition.

4. The "Complexity Score" (Quantitative Bounds)

Finally, the paper provides a way to measure "How hard is this shape to build?"

In the past, mathematicians thought that as you add more dimensions (like moving from a 2D drawing to a 3D model to a 10D hyper-shape), the task became exponentially harder—the "curse of dimensionality."

Lee’s math suggests something much more optimistic. He shows that the difficulty of building a shape doesn't necessarily depend on how many dimensions there are, but rather on the "Fischer Norm"—which you can think of as the "wiggliness" or the "complexity of the curves" of the shape.

If a shape is relatively smooth, even in a thousand dimensions, a two-layer neural network can build it surprisingly easily.

Summary: Why does this matter?

If we want to understand why AI works, we need to know why it can learn complex patterns so quickly. This paper provides a mathematical bridge. It shows that neural networks aren't just "guessing" at patterns; they are performing a sophisticated form of "mathematical construction," using simple ramps to build complex worlds, and it gives us the blueprints to understand exactly how they do it.

Technical Summary: Explicit Integral Representations and Quantitative Bounds for Two-Layer ReLU Networks

Author: Anthony Lee (University of Bristol)
Date: April 28, 2026

1. Problem Statement

While empirical evidence suggests that neural networks are highly capable of approximating high-dimensional functions, a robust theoretical explanation for this capacity—especially regarding the specific mechanics of ReLU activations—remains an open challenge. Existing theoretical frameworks (such as Barron’s) often rely on Fourier-based approaches, which can be difficult to translate into explicit, interpretable representations for specific activation functions like ReLU.

This paper addresses the problem of finding explicit integral representations for multivariate polynomials within two-layer ReLU networks and establishing quantitative error bounds that do not suffer from the "curse of dimensionality" (i.e., bounds that do not depend explicitly on the dimension $d$ or degree $k$ in a way that explodes).

2. Methodology

The author shifts the perspective from Fourier transforms to the study of an integral operator specifically tailored to the ReLU function. The methodology follows these core steps:

The ReLU Operator ( $T_\mu$ ): The paper defines an operator $T_\mu g(x) = \int g(\langle x, v \rangle v) \mu(dv)$ . The goal is to find a "ReLU representative" $g$ such that $f = T_\mu g$ .
Spectral Analysis: By utilizing the Funk–Hecke formula, the author demonstrates that for spherically symmetric distributions $\mu$ , the operator $T_\mu$ is bijective on the space of polynomials. This allows for the construction of a basis using eigenfunctions (harmonic polynomials).
Harmonic Extension & Lifting: To simplify the representations, the author employs a "lifting" technique. A polynomial $f \in \mathbb{R}^d$ is extended to a harmonic polynomial $H(f) \in \mathbb{R}^{d+1}$ . This allows the network to represent the function using "sharpened" components.
The Heat Equation Connection: The author introduces a "sharpened" representation where the homogeneous components of the polynomial are processed via the inverse heat equation (running the heat equation backwards in time). This "denoises" the components, leading to more efficient representations.
Optimization: The paper uses the principle that a random two-layer network approximates an integral representation. By choosing an optimal probability measure $\pi$ to minimize the expected $L_2(D)$ error, the author derives a deterministic bound for the network's performance.

3. Key Contributions

Explicit Construction: Unlike Fourier methods, this paper provides a direct, constructive method to write any multivariate polynomial as an integral of ReLU neurons.
Sharpened Representation (Theorem 23): The derivation of a specific integral representation involving $f^\sharp_k = \exp(-\frac{\sigma^2}{2}\Delta)f_k$ . This links ReLU networks to the theory of Hermite polynomials and the heat equation.
Dimension-Independent Bounds: The development of error bounds that depend on the Fischer norm of the polynomial's components rather than the raw dimension or degree.
Harmonic Lifting Framework: A method to transform complex polynomial approximation problems into simpler harmonic ones by increasing the dimensionality by one.

4. Key Results

Existence and Uniqueness: For any spherically symmetric $\mu$ , the operator $T_\mu$ is invertible, ensuring every polynomial has a unique ReLU representative.
Quantitative Error Bound (Theorem 30): The paper proves that for a sub-Gaussian distribution $D$ , there exists a two-layer ReLU network $f_n$ such that:
$\|f_n - f\|_{L_2(D)} \leq \frac{1}{\sqrt{n}} \sum_{k \geq 1} C_{d,k} \|f_k\|_F$
Crucially, if the distribution $D$ scales appropriately (e.g., $D \sim N(0, \frac{1}{d}I)$ ), the constants $C_{d,k}$ become dimension-independent. This implies that "regular" high-dimensional functions are surprisingly easy for two-layer networks to approximate.
Empirical Consistency: Numerical observations suggest that the "sharpened" theoretical representation (using negative biases and specific scaling) aligns closely with the behavior of networks trained via standard optimization (like He initialization).

5. Significance

This work provides a significant theoretical bridge between classical analysis (harmonic polynomials, heat equations) and deep learning theory.

Complexity Measures: It introduces the Fischer norm as a new, potentially more accurate measure of "function difficulty" for neural networks than the standard Barron norm.
Theoretical Insight: It explains why neural networks can avoid the curse of dimensionality: if a function's importance is spread across components (low Fischer norm relative to dimension), the error remains controlled even as $d \to \infty$ .
Future Directions: The paper lays the groundwork for analyzing deeper networks by treating layers as a composition of integral operators, potentially offering a path toward understanding the representational power of deep architectures.