🔢 mathematics

Sharp Sobolev Sandwich and Approximation Rates of Radon-Domain $L^p$ Ridge Integral Spaces for ReLU $^k$ Networks

This paper establishes that the Radon-domain $L^p$ space of functions representable by shallow $\mathrm{ReLU}^k$ networks forms a sharp Sobolev sandwich around the critical regularity space $H^{k+(d+1)/2}$ , with the gap determined by the Seeger–Sogge–Stein loss, and leverages this theory to derive optimal $L^p$ approximation rates for discretized neural networks.

Original authors: Juncai He, Zitong Tian

Published 2026-06-24

📖 4 min read🧠 Deep dive

CC BY 4.0

Original authors: Juncai He, Zitong Tian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a complex 3D sculpture (a mathematical function) using only simple, flat sheets of material. In the world of machine learning, these "sheets" are called neurons, and the way they are stacked together is called a neural network.

This paper is like a master blueprint that explains exactly how well you can build any shape using a specific type of sheet called ReLUk. The "k" just means the sheet can be bent or folded $k$ times (making it smoother or more flexible).

Here is the breakdown of their discovery, using simple analogies:

1. The Problem: How Many Sheets Do You Need?

For a long time, we knew you could build almost any shape with enough sheets (neurons). But we didn't know how many you needed to get a specific level of detail.

The Question: If I want my sculpture to be smooth and accurate, do I need 10 sheets, 1,000, or 1,000,000?
The Goal: The authors wanted to find the exact "recipe" for the smoothest possible shapes and the most efficient way to build them.

2. The Secret Ingredient: The "Radon Domain"

To solve this, the authors didn't look at the sculpture from the front. Instead, they looked at it through a magical lens called the Radon Transform.

The Analogy: Imagine taking a loaf of bread and slicing it into thin pieces from every possible angle. The Radon Transform is the collection of all those 2D slices.
The Discovery: The authors realized that if you look at the "slices" (the Radon domain) rather than the whole loaf, the math becomes much clearer. They defined a special "space" (a library of functions) based on how smooth these slices are. They call this the Radon-domain $L^p$ space.

3. The "Sandwich" Discovery

This is the paper's biggest "Aha!" moment.

The Perfect Case ( $p=2$ ): When looking at the problem in a specific mathematical way (like measuring average error), the library of shapes you can build with these neurons is exactly the same as a famous class of smooth shapes known as Sobolev spaces. It's a perfect match, like two puzzle pieces fitting together with no gaps.
The General Case ( $1 < p < \infty$ ): When you change the way you measure the error (looking at different types of "roughness"), the perfect match turns into a Sandwich.
- The Bread (Top): A slightly smoother class of shapes.
- The Bread (Bottom): A slightly rougher class of shapes.
- The Filling: The shapes your neural network can actually build.
- The Gap: The authors calculated the exact size of the gap between the top and bottom bread. This gap is caused by a known mathematical "friction" (called the Seeger–Sogge–Stein loss) that happens when you slice and reassemble the data. It's the unavoidable cost of turning slices back into a loaf.

4. Why Does This Matter? (The Approximation Rate)

Now that they know exactly what kind of shapes these networks can build, they can predict how fast the network learns.

The Recipe: They showed that if you pick your neurons randomly (like grabbing random slices of bread) but in a smart, uniform way, you can build a very accurate sculpture very quickly.
The Result: They proved that for the smoothest shapes, the error drops at the fastest possible speed allowed by math.
- If you double the number of neurons, the error doesn't just go down a little; it drops at a specific, optimal rate.
- They also showed how to remove a small "logarithmic" penalty that previous methods had, making the process even more efficient.

Summary in Plain English

Think of the authors as architects who finally figured out the exact physics of building with "ReLU" bricks.

They found a special way to look at the bricks (the Radon domain) that reveals their true potential.
They proved that for the most common measurement, these bricks can build exactly the smoothest possible structures.
For other measurements, they proved the structures fit perfectly between two known limits (the "Sandwich"), with the gap size being mathematically unavoidable.
Finally, they showed that by using a simple, random sampling method, you can build these structures with the maximum possible speed and efficiency, proving that these simple networks are incredibly powerful tools for learning smooth patterns.

In short: They didn't just say "neural networks work." They wrote the exact instruction manual for how well they work, why they work, and how fast they can learn, using a clever mathematical lens to see the hidden structure of the data.

Technical Summary: Sharp Sobolev Sandwich and Approximation Rates of Radon-Domain $L^p$ Ridge Integral Spaces for ReLU $^k$ Networks

1. Problem Statement

The paper addresses the quantitative approximation theory for shallow neural networks with ReLU $^k$ activations ( $\sigma_k(x) = \max\{0, x\}^k$ ). While qualitative universal approximation theorems are well-established, determining the precise number of neurons required to approximate functions in specific target classes (such as Sobolev spaces) with a given accuracy remains a central challenge.

Specifically, the authors seek to:

Characterize the function space generated by ridge integral representations with $L^p$ coefficient densities in the Radon domain.
Establish the precise relationship between these Radon-domain spaces and classical Sobolev spaces $W^{s,p}(\Omega)$ .
Derive sharp approximation rates for discretizing these integral representations using deterministic interpolation and random sampling, particularly in the $L^p$ regime ( $1 < p < \infty$ ) and the Hilbert case ( $p=2$ ).

2. Methodology

The authors employ a framework based on the Radon transform and Fourier integral operator (FIO) theory.

Radon-Domain Representation: The core object is the space $RL^p_k(\Omega)$ , defined as the image of the ridge integral operator $S^k_R$ acting on $L^p$ densities $g(w, b)$ over a bounded bias window $M_R = S^{d-1} \times [-R, R]$ . The integral is given by:
$S^k_R g(x) = \int_{M_R} \sigma_k(w \cdot x - b) g(w, b) \, d\lambda(w, b)$
Analysis and Synthesis Operators: The paper defines an analysis operator $A_k$ (involving the Radon transform, a ramp filter, and $(k+1)$ -th order derivatives) and a synthesis operator $S^k_R$ . The relationship between these operators and the canonical density $g_u = A_k u$ is central to the theory.
Fourier Analysis ( $p=2$ ): For the Hilbert case, the authors use elementary Fourier analysis and the Plancherel theorem to establish an exact isometry between the Radon-domain $L^2$ space and the critical Sobolev space $H^{s_k}(\Omega)$ , where $s_k = k + \frac{d+1}{2}$ .
FIO Regularity ( $1 < p < \infty$ ): For general $p$ , the exact isometry fails. The authors utilize the theory of Fourier integral operators, specifically the Seeger–Sogge–Stein theorem, to analyze the regularity of the Radon transform and its dual as operators on $L^p$ spaces. This introduces a "loss" term $\delta_p = (d-1)|\frac{1}{p} - \frac{1}{2}|$ .
Discretization: The integral representation is discretized using a combination of a deterministic interpolation skeleton and uniform random sampling of neurons to derive probabilistic approximation bounds.

3. Key Contributions and Results

A. The Radon-Domain $L^p$ Space and Sobolev Sandwich
The paper defines the space $RL^p_k(\Omega)$ and proves a "Sobolev sandwich" embedding theorem. Unlike the $p=2$ case where the space coincides exactly with a Sobolev space, for $p \neq 2$ , the Radon-domain space is sandwiched between two Sobolev spaces with a sharp gap determined by the Seeger–Sogge–Stein loss.

Theorem 3.3 (Sobolev Sandwich):
$W^{s_{k,p}, p}(\Omega) \hookrightarrow RL^p_k(\Omega) \hookrightarrow W^{r_{k,p}, p}(\Omega)$
where the smoothness exponents are:
$s_{k,p} = k + \frac{d+1}{2} + \delta_p, \quad r_{k,p} = k + \frac{d+1}{2} - \delta_p$
and $\delta_p = (d-1)|\frac{1}{p} - \frac{1}{2}|$ $δ_{p} = (d - 1) ∣ \frac{1}{p} - \frac{1}{2} ∣$ .
- When $p=2$ , $\delta_2 = 0$ , and the sandwich collapses to the exact identity $RL^2_k(\Omega) = H^{s_k}(\Omega)$ .
- The gap $s_{k,p} - r_{k,p} = 2\delta_p$ is proven to be optimal; no continuous embedding exists with a smaller gap.

B. Exact $L^2$ Isometry
For $p=2$ , the paper provides an elementary Fourier analysis proof (Theorem 2.2) showing that $RL^2_k(\Omega)$ is exactly the critical Sobolev space $H^{s_k}(\Omega)$ with equivalent norms. This recovers and clarifies previous results regarding the Reproducing Kernel Hilbert Space (RKHS) of ReLU $^k$ networks, explicitly separating the weight and bias parameters on a cylinder rather than a sphere.

C. Approximation Rates
The authors discretize the integral representation to obtain high-probability approximation rates for linearized neural networks.

Theorem 4.1: For any $u \in RL^p_k(\Omega)$ , there exist coefficients such that the approximation error in $L^p(\Omega)$ decays as:
$\|u - f_n\|_{L^p} \leq C (\log(2/\delta))^{\alpha_p} \|u\|_{RL^p_k} n^{-\alpha_p - \gamma_{k,p}}$
where $\gamma_{k,p} = \frac{k+1/p}{d}$ and $\alpha_p = 1 - \frac{1}{\min(p, 2)}$ .
Optimal Hilbert Rate: At $p=2$ , the rate simplifies to $O(n^{-s_k/d})$ , recovering the optimal information-theoretic rate for the critical Sobolev space $H^{s_k}(\Omega)$ without the logarithmic factors present in some prior random feature analyses.

4. Significance and Claims

The paper claims to provide a unified and sharp theoretical framework for understanding the approximation capabilities of shallow ReLU $^k$ networks through the lens of the Radon transform.

Clarification of Regularity: The work clarifies how the regularity of the activation function (contributing $k+1$ derivatives) and the back-projection of the Radon transform (contributing $\frac{d-1}{2}$ derivatives in $L^2$ , or $\frac{d-1}{2} - \delta_p$ in $L^p$ ) jointly determine the critical smoothness of the representable functions.
Sharpness of Bounds: By identifying the Seeger–Sogge–Stein loss as the exact mechanism for the gap in the $L^p$ regime, the paper establishes that the derived Sobolev sandwich is sharp. This resolves the question of how much regularity is lost when moving from the Hilbertian ( $p=2$ ) setting to general $L^p$ settings.
Optimal Rates: The derived approximation rates for $p=2$ match the known optimal rates for shallow networks, confirming that linearized networks with fixed, well-distributed parameters can achieve the information-theoretic limit for the critical Sobolev space.

The authors emphasize that their results are derived using elementary Fourier analysis for the $L^2$ case and established FIO theory for the $L^p$ case, avoiding the need for more complex machinery while providing precise constants and sharp exponents. The work serves to bridge the gap between integral representation theory, harmonic analysis, and the quantitative theory of neural network approximation.