🤖 machine learning

Optimal Neural Network Approximation via Empirical Least Squares with Deterministic Samples

This paper establishes a rigorous theory for approximating solutions to elliptic spectral equations on the sphere using linearized ReLU $^k$ neural networks via empirical least squares with deterministic samples, proving optimal convergence rates and deriving key Bernstein inequalities for the associated network spaces.

Original authors: Xinliang Liu, Tong Mao, Jinchao Xu

Published 2026-08-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xinliang Liu, Tong Mao, Jinchao Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the world. In the realm of artificial intelligence, the "brain" of this robot is a neural network, a complex web of mathematical functions designed to recognize patterns. One of the most popular tools in this toolkit is the "ReLU" function, which acts like a simple switch: if a signal is positive, it passes it through; if it's negative, it cuts it off to zero. While these switches are great for building deep, powerful networks, mathematicians have long struggled to prove exactly how well they work when we try to solve specific, tricky equations, especially when we only have a limited number of data points to train on.

This paper dives into a specific corner of that puzzle: solving equations that describe smooth, wave-like phenomena on the surface of a sphere (like the Earth or a ball). The researchers are asking a fundamental question: If we use a neural network made of these ReLU switches to approximate a solution, and we only check the network's performance at a few specific spots (samples) rather than everywhere, will it still get the right answer? They are particularly interested in "deterministic" sampling, where we carefully pick our test spots, rather than just throwing darts at a board randomly. Understanding this is crucial because in the real world, we rarely have infinite data; we need to know how many samples are enough to guarantee a good result without wasting time or computing power.

The authors of this paper have developed a rigorous mathematical theory that acts like a safety net for these neural network approximations. They prove that if you arrange your network's "knobs" (parameters) in a specific, well-spaced pattern on a sphere, and you pick your test points carefully, the network will converge to the correct solution at the fastest possible speed. Think of it like tuning a radio: if you turn the dial just right (the optimal arrangement of parameters) and listen at the right stations (the collocation points), you get a crystal-clear signal. The paper shows that you don't need a million samples to get this clarity; you only need a number of samples roughly equal to the number of adjustable knobs in your network. This is a big deal because it means the method is efficient and doesn't require an impossible amount of data.

However, the paper is very careful about what it claims. It proves that this "perfect" efficiency works specifically on the surface of a sphere and for a specific type of equation involving these ReLU switches. The authors explicitly state that you cannot simply swap the sphere for a flat box (like a cube) and expect the same magic to happen immediately. Their theory for the sphere doesn't automatically guarantee the same results for flat, bounded domains like the walls of a room or a computer screen. While they do show how to mathematically "lift" a problem from a flat box onto a sphere to use their new theory, they admit that this is a clever workaround for a specific case, not a universal fix for all shapes. Furthermore, while they provide strong mathematical proofs for the sphere, their results for flat domains are currently just numerical experiments—simulations that look promising but haven't been rigorously proven to work in the same way yet.

The core of their discovery relies on a new mathematical tool they call a "Bernstein inequality." In simple terms, this is a rule that limits how "wiggly" or chaotic a neural network can get. It's like saying, "If you know the average height of a wave, you can't suddenly have a spike that is a million times taller unless you have a lot of space to do it." This rule allows the authors to prove that the error in their approximation is tightly controlled. They also show that if you pick your test points randomly (like throwing darts), you can still get a good answer most of the time, but you might need a few more points to be safe, and there's a tiny chance of a bad outcome.

In their experiments, the researchers tested their theory on spheres of different sizes and with different types of ReLU switches. They found that the errors dropped exactly as their theory predicted, confirming that the method works beautifully in the controlled environment of a sphere. When they tried to apply the same logic to a flat cube, the errors dropped, but not as fast as the theory predicted for the sphere, reinforcing their warning that the sphere and the cube are mathematically different beasts. Ultimately, this paper provides a solid, proven foundation for using neural networks to solve equations on spherical surfaces, offering a clear roadmap for how many samples are needed to get a precise answer, while humbly acknowledging that the journey to applying this to every shape in the universe is still ongoing.

Technical Summary: Optimal Neural Network Approximation via Empirical Least Squares with Deterministic Samples

Problem Statement
This work addresses the theoretical analysis of discrete residual least-squares approximation for elliptic spectral equations $L_\beta u = f$ on the unit sphere $S^d$ . The study focuses on linearized ReLU $^k$ neural networks, where the activation function is $\sigma_k(x) = (\max\{0, x\})^k$ for $k \in \mathbb{N}$ . The primary challenge lies in bridging the gap between continuous approximation theory and discrete empirical risk minimization. Specifically, the authors investigate whether optimal approximation rates, typically derived for continuous function spaces, can be recovered using a finite number of deterministic or random collocation points without numerical integration (quadrature). The work considers the case where the hidden layer parameters (weights and biases) are fixed on the sphere, reducing the problem to a linear least-squares optimization over the output coefficients.

Methodology
The authors develop a rigorous framework based on the geometry of the parameter space and the spectral properties of the network functions. Key methodological components include:

Spherical Normalization: Leveraging the positive homogeneity of $\sigma_k$ , the authors normalize the parameter space to the sphere $S^d \subset \mathbb{R}^{d+1}$ . The network functions take the form of ridge functions $\sigma_k(\theta \cdot \eta)$ where $\theta, \eta \in S^d$ .
Bernstein Inequalities: A central analytical tool is the derivation of a Bernstein inequality for linearized ReLU $^k$ network spaces. This inequality bounds higher-order Sobolev norms by lower-order norms, scaled by the inverse of the antipodal separation distance $h$ of the network parameters. Specifically, for $0 \le s < r < k + 1/2$ , the inequality $\|v_n\|_{H^r(S^d)} \lesssim h^{-(r-s)} \|v_n\|_{H^s(S^d)}$ is established.
Residual Space Analysis: The authors extend these inequalities to the residual space $V_{n,\beta} = L_\beta L^k_n(\Theta_n)$ , where $L_\beta$ is a positive elliptic spectral multiplier of order $\beta$ . This allows for stability estimates in the context of solving PDEs.
Sampling Inequalities: The discrete least-squares problem is analyzed using Marcinkiewicz–Zygmund inequalities and sampling estimates to compare empirical norms (computed at collocation points) with continuous Sobolev norms.
Extension to Bounded Domains: For the order-zero case ( $\beta=0$ ), the authors construct an affine-network interpretation on bounded Lipschitz domains by lifting the target function to the sphere, applying the spherical theory, and restricting the result back to the domain. This relies on constructive Sobolev extension operators and homogeneous transformations.

Key Contributions and Results

Optimal Deterministic Rates: The paper proves that for antipodally quasi-uniform network parameter sets and quasi-uniform collocation points with sample size $m \gtrsim n$ , the discrete residual least-squares solution $u_{n,m}$ achieves the optimal approximation rate. Specifically, if $k > \beta + \frac{d-1}{2}$ and the target function $f$ has sufficient smoothness, the error satisfies:
$\|u - u_{n,m}\|_{H^{s+\beta}(S^d)} \lesssim n^{-\frac{r-s}{d}} \|f\|_{H^r(S^d)}$
This rate matches the best possible rate for continuous approximation, demonstrating that the discretization error does not degrade the convergence order when using deterministic samples.
High-Probability Random Estimates: For independent and identically distributed (i.i.d.) uniformly distributed collocation points, the authors establish a high-probability residual estimate. The rate is near-optimal, incurring only a logarithmic factor and an arbitrarily small loss in smoothness ( $\epsilon$ ).
Inverse Approximation Theorem: As a consequence of the Bernstein inequality, the paper establishes an inverse theorem: if a function can be approximated by linearized ReLU $^k$ networks at a certain rate, the function possesses corresponding Sobolev smoothness.
Affine Network Interpretation: For $\beta=0$ , the spherical results are translated to bounded domains $\Omega \subset \mathbb{R}^d$ . The authors show that a spherical linearized network restricted to a spherical cap induces an ordinary affine ReLU $^k$ network. This provides a constructive path to approximate functions on $\Omega$ using the spherical theory, though the authors note this does not constitute a direct stability proof for least squares formed solely from samples in $\Omega$ .

Significance and Claims
The paper claims to provide the first rigorous approximation results of the form (1.7) for linearized neural network spaces under discrete least-squares discretization, even with equal-weight sampling. The authors emphasize that their approach bypasses the numerical integration step, which is often a source of bias and instability in neural network-based PDE solvers (such as Physics-Informed Neural Networks).

The significance lies in:

Stability Mechanism: The derivation of the Bernstein inequality for ReLU $^k$ spaces on the sphere serves as the fundamental stability mechanism, controlling the residual space and ensuring that the discrete problem is well-posed and convergent.
Sample Complexity: The results demonstrate that the sample size required for optimal convergence is proportional to the number of trainable output coefficients ( $m \asymp n$ ), which is a favorable sample complexity compared to methods requiring significantly denser sampling to overcome quadrature errors.
Theoretical Foundation: The work offers a mathematically rigorous foundation for discrete least-squares training of neural networks, moving beyond probabilistic generalization bounds to deterministic error estimates based on the geometry of the parameter set.

Limitations and Scope
The authors explicitly state that their bounded-domain results are restricted to the order-zero case ( $\beta=0$ ). They do not claim to have solved the stability of least-squares formulations for general elliptic PDEs ( $\beta > 0$ ) on bounded domains with boundary conditions, noting that such a theory would require handling boundary residuals and variable-coefficient operators, which remains an open direction for future work. The numerical experiments are presented as finite-range evidence supporting the theoretical predictions rather than empirical proofs of asymptotic behavior.

More like this