🔢 mathematics

Sharp Sobolev Approximation on General Domains by Linearized Shallow Networks with Analytic Activations

This paper establishes that linearized shallow neural networks with analytic activations and fixed, quasi-uniform parameter sets achieve sharp Sobolev approximation rates on general domains, offering a more practical alternative to previous finite-difference constructions by avoiding the need for extremely small parameter scales.

Original authors: Jia Li, Tong Mao, Jinchao Xu

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jia Li, Tong Mao, Jinchao Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of modern computing, artificial intelligence relies on mathematical structures known as neural networks to learn patterns from data. Imagine these networks as vast, flexible webs of simple processing units that can be tuned to mimic almost any shape or function. A common and efficient version of this web is the "shallow" network, which uses just one layer of these processing units to transform an input into an output. The power of such a system depends heavily on how well it can approximate complex, smooth curves found in the real world, a concept mathematicians describe using a measure of smoothness called Sobolev approximation. For decades, researchers have known that these networks can indeed learn these curves, but a critical question remained: how efficiently can they do it if the internal settings of the network are fixed in advance, rather than being custom-tailored for every single new problem?

This question matters because in many practical applications, we want to use a pre-made, reliable set of network settings that works well for a whole class of problems without needing to retrain the entire system from scratch. If the settings are chosen poorly, the network might require an enormous number of units to achieve a decent result, making it slow and expensive. If they are chosen wisely, the network can achieve high accuracy with far fewer resources. The challenge lies in finding a specific arrangement of these internal settings that guarantees the best possible performance for smooth functions, regardless of the specific function being studied.

A team of researchers has now solved this problem for a broad and important category of activation functions, which are the mathematical rules that determine how a network unit responds to input. They demonstrated that by carefully selecting the internal parameters of a shallow network using a specific, structured pattern, one can achieve the fastest possible rate of accuracy improvement as the network grows. Their work proves that for a wide range of smooth functions, a network with a fixed set of internal settings can approximate the target function with an error that shrinks at the optimal mathematical rate as the number of units increases. This is a significant achievement because it moves beyond theoretical possibilities to provide a concrete, reliable blueprint for building efficient networks that do not need to be re-engineered for every new task.

The researchers focused on a specific type of network where the internal "knobs"—the numbers that shift and scale the input before it is processed—are set independently of the specific function the network is trying to learn. In previous attempts to solve this, researchers often relied on methods that required these internal knobs to be clustered extremely close together, like a dense crowd of people standing shoulder-to-shoulder. While mathematically valid, such tight clustering creates practical difficulties for computers, as it can lead to numerical instability and make the system hard to use. The new approach avoids this pitfall entirely. Instead of forcing the parameters into a tight, fragile cluster, the researchers designed a set of parameters that are spread out evenly across a fixed, stable range. This distribution is based on a mathematical pattern known as quasi-Chebyshev, which ensures that the points are spaced in a way that maximizes their coverage and minimizes gaps, much like how a well-planned grid of sensors would cover a field more effectively than a random scattering.

The core of their discovery lies in a one-dimensional construction that serves as the foundation for the entire system. They proved that for a class of smooth, analytic functions, using these evenly spaced parameters allows the network to capture the essential features of a target function with remarkable precision. The researchers showed that this method works for several common activation functions, including the hyperbolic tangent and the sigmoid function, which are staples in neural network design. By establishing that these fixed parameter sets can achieve the sharpest possible approximation order, they confirmed that the network's error decreases at the fastest rate theoretically possible as the number of units grows. This means that for a given level of smoothness in the target function, the network gets more accurate at the optimal speed, without needing to adjust its internal settings for each new problem.

To extend this success from a single line to complex, multi-dimensional spaces, the team combined their one-dimensional result with a powerful mathematical tool known as a lifting theorem. This theorem allows the properties of a one-dimensional approximation to be lifted into higher dimensions, effectively building a multi-dimensional network from the simpler, one-dimensional building blocks. By using a specific arrangement of directions that are evenly distributed across a sphere, they constructed a multi-dimensional network that retains the optimal accuracy of the one-dimensional case. The result is a network architecture where the internal parameters are fixed, the directions are evenly spread out, and the bias terms follow the stable, quasi-Chebyshev pattern. This combination ensures that the network can handle high-dimensional data with the same efficiency and stability as its one-dimensional counterpart.

The significance of this work is that it provides a definitive answer to the question of how to set up a linearized shallow network for optimal performance. The researchers explicitly showed that their method is superior to previous approaches that relied on finite-difference constructions, which often required the internal parameters to be scaled down to such a tiny degree that they became impractical for real-world computation. In contrast, the new parameter sets remain distributed over fixed intervals, making them robust and amenable to practical calculation. The paper proves that this approach is not just a theoretical curiosity but a viable path forward for constructing efficient, pre-fabricated neural networks. By demonstrating that the optimal rate of approximation can be achieved with fixed, well-distributed parameters, the study offers a clear and reliable method for designing neural networks that are both powerful and computationally stable, paving the way for more efficient artificial intelligence systems in the future.

Technical Summary: Sharp Sobolev Approximation on General Domains by Linearized Shallow Networks with Analytic Activations

Problem Statement
The paper addresses the quantitative approximation of functions in Sobolev spaces $H^r(\Omega)$ on bounded domains $\Omega \subset \mathbb{R}^d$ using linearized shallow neural networks. In the linearized setting, the inner parameters (weights and biases) are fixed a priori, independent of the target function $f$ , and only the outer coefficients are optimized. The goal is to construct a fixed feature space $\mathcal{L}_n$ of dimension $n$ such that the best approximation error satisfies the sharp rate:
$\inf_{g \in \mathcal{L}_n} \|f - g\|_{L^2(\Omega)} \lesssim n^{-r/d} \|f\|_{H^r(\Omega)}$
for all $r > 0$ . While optimal rates are known for homogeneous activations like ReLU $^k$ via radial scaling arguments, extending these results to non-homogeneous analytic activations (e.g., $\tanh$ , sigmoid) on general Euclidean domains has been challenging. Previous constructions, such as the finite-difference method in [32], often require parameter scales that decay factorially with the resolution, leading to numerically unstable, clustered feature sets. This work seeks to establish sharp approximation rates for analytic activations using parameter sets that remain distributed over fixed intervals.

Methodology
The authors employ a three-stage methodology to bridge one-dimensional approximation theory with high-dimensional linearized networks:

One-Dimensional Construction: The core technical step is a sharp Jackson-type estimate for analytic activations on a bounded interval. The authors utilize quasi-Chebyshev parameter sets (bi-Lipschitz images of Chebyshev nodes) for the biases. They prove that for analytic activations satisfying a quantitative non-cancellation condition on their Taylor coefficients (specifically, $|\sigma^{(k)}(0)|/k! \geq C \rho^{-k}$ ), the span of shifted activation functions achieves the optimal Sobolev approximation order $m^{-(r-s)}$ using $m$ features.
- For $\tanh$ , the authors bypass the Taylor coefficient condition by utilizing a rational transformation. They map the problem to a space spanned by rational functions of the form $1/(1+\omega t)$ , where the parameters $\omega$ form a quasi-Chebyshev set, allowing the application of the analytic framework.
Ridge-Function Lifting: To extend the univariate result to $d$ dimensions, the paper utilizes a ridge-function lifting theorem. Building on the work in [37], the authors generalize the lifting argument to arbitrary quasi-uniform direction sets on the sphere $S^{d-1}$ . This theorem states that if a univariate space $X_m$ approximates functions on an interval with rate $m^{-q}$ , then the tensor-product space generated by ridge functions with directions from a quasi-uniform set approximates functions on the domain with rate $m^{-(q + (d-1)/2)}$ .
Interpolation and Real Interpolation: The proofs rely on least-squares interpolation operators constructed on quasi-Chebyshev points. The authors establish Marcinkiewicz–Zygmund inequalities for cosine polynomials sampled at these points to ensure the stability of the interpolation. To handle the full range of smoothness $r > 0$ (including cases where $r \leq (d-1)/2$ ), the authors employ real interpolation between the $L^2$ estimate and the high-smoothness Sobolev estimate derived from the lifting theorem.

Key Contributions and Results

Sharp Rates for Analytic Activations: The paper proves that for a class of analytic activations (including $\tanh$ , sigmoid, arctan, and specific rational functions) satisfying a non-cancellation condition on their Taylor coefficients, linearized shallow networks with quasi-Chebyshev biases and quasi-uniform directions achieve the sharp approximation rate $n^{-r/d}$ in the $L^2$ norm.
Fixed-Interval Parameter Geometry: A primary contribution is the construction of parameter sets that remain distributed over fixed intervals with algebraic separation. This contrasts with the finite-difference construction in [32], which requires parameter scales to shrink factorially ( $h_n \sim \exp(-m \log m)$ ) to satisfy admissibility conditions, leading to "factorially clustered" features. The proposed quasi-Chebyshev sets avoid this clustering, making them more amenable to practical computation.
Generalization of Lifting Theorems: The work extends the ridge-function lifting theorem from specific cubature points to arbitrary quasi-uniform direction sets, demonstrating that the optimal rate depends only on the resolution of the univariate parameters and the quasi-uniformity of the directions.
Verification of Conditions: The authors verify the required non-cancellation conditions for several specific activations. For $\tanh(x + q\pi/2)$ with rational $q \notin \{0, \pm 1\}$ , they use Diophantine approximation properties of algebraic numbers (specifically Niven's Theorem and the Gelfond–Schneider theorem) to show that the Taylor coefficients do not vanish and satisfy the necessary lower bounds.

Significance and Claims
The paper claims to resolve the construction of sharp linearized approximation spaces for analytic activations on general domains without relying on the homogeneity of the activation function (which is unavailable for $\tanh$ or sigmoid). By reducing the multidimensional problem to a sharp one-dimensional Jackson estimate and a product-type lifting, the authors demonstrate that the optimal rate $n^{-r/d}$ is attainable with prescribed parameter sets.

The significance lies in the geometric stability of the proposed parameter sets. Unlike previous methods that require extreme parameter scaling to achieve theoretical rates, this work provides a construction where parameters are distributed over fixed intervals. The authors note that while the approximation exponent and the class of smooth activations are not new in the broader literature, the specific contribution is the prescribed quasi-Chebyshev feature geometry on fixed intervals and its successful lifting through arbitrary quasi-uniform directions. The paper concludes by identifying the quantitative conditioning of the resulting feature matrices as an open question for future investigation, maintaining a modest scope regarding numerical implementation details.

Technical Summary: Sharp Sobolev Approximation on General Domains by Linearized Shallow Networks with Analytic Activations

More like this