Optimal Sobolev Approximation by Deterministic and Random Shallow Sigmoidal Networks

In the vast landscape of modern mathematics, there is a persistent question about how well we can capture the shape of complex reality using simple building blocks. Imagine trying to describe a rugged mountain range, a swirling weather pattern, or the flow of electricity through a circuit. These phenomena are defined by their smoothness and their roughness, their gentle slopes and their sharp edges. Mathematicians call this quality "regularity." For decades, researchers have relied on artificial neural networks—computational models inspired by the human brain—to approximate these complex functions. Specifically, they use "shallow" networks, which have a single layer of hidden processing units sandwiched between an input and an output. These units often use a smooth, S-shaped curve, known as a sigmoid, to transform data. The central puzzle has been: if we fix the internal settings of these units in advance, either by carefully choosing them or by picking them at random, can the network still learn to mimic any smooth function with high precision? The answer determines whether these flexible tools are merely heuristic tricks or mathematically rigorous instruments capable of solving the hardest equations in physics and engineering.

A team of researchers has now settled this question for a broad class of these smooth, S-shaped curves. They proved that shallow networks can indeed achieve the best possible rate of accuracy for approximating smooth functions, provided the network has enough units. This holds true whether the internal settings are chosen by a precise, deterministic recipe or are drawn randomly from a pool of possibilities. The researchers focused on standard activation functions used in practice, such as the hyperbolic tangent and the error function, which are known for their smooth, bell-like derivatives. Their work demonstrates that with a specific number of hidden units, the network can approximate a target function with an error that shrinks predictably as more units are added. This rate of improvement is not just good; it is mathematically optimal, meaning no other method using the same amount of computational resources could possibly do better.

The study distinguishes between two ways of setting up these networks. In the first approach, the researchers constructed a deterministic dictionary of features. They carefully selected the directions and offsets for each hidden unit, arranging them like a precise grid to cover the space of possible inputs. They showed that for any function with a certain level of smoothness, this carefully built network could approximate it with an error that decreases at the fastest possible speed allowed by the laws of mathematics. This speed depends on the dimension of the problem and the smoothness of the target function. If the function is very smooth, the error drops quickly; if the function is rougher, the drop is slower, but it follows a perfect, predictable algebraic rule. Crucially, the researchers also proved that the numbers used to combine these units do not grow uncontrollably, ensuring the method remains stable and usable.

In the second, perhaps more surprising, approach, the researchers tested what happens when the internal settings are not chosen carefully but are instead sampled randomly. In many practical applications, engineers prefer random sampling because it is faster and easier to implement than designing a perfect grid. The researchers proved that even with random sampling, the network retains the same powerful ability to approximate smooth functions. As long as the random selection comes from a distribution that covers the necessary space without leaving large gaps, the network will, with very high probability, achieve the same optimal rate of accuracy. The only cost for this randomness is a small, logarithmic increase in the number of units required to reach the same level of precision. This finding is significant because it validates the use of random features in high-dimensional problems, confirming that the "luck" of random sampling does not come at the expense of mathematical power.

To verify their theoretical proofs, the researchers ran extensive numerical experiments across a wide range of scenarios. They tested dimensions ranging from two to ten, targeting functions with varying degrees of smoothness and measuring errors in different ways, from simple average differences to more complex measures involving derivatives. In every case, the computer simulations matched their mathematical predictions perfectly. The graphs of error versus the number of units showed straight lines on a logarithmic scale, confirming that the error was shrinking at the exact algebraic rates the theory predicted. Whether the features were deterministic or random, whether the target was a simple curve or a complex ten-dimensional surface, the results were consistent. The experiments covered a broad spectrum of conditions, including high-dimensional settings where intuition often fails, and in every instance, the networks performed exactly as the new theory described.

The implications of this work extend beyond the abstract realm of approximation theory. It provides a solid foundation for using fixed-feature neural networks in solving partial differential equations, which are the language of physics, engineering, and finance. These equations often describe systems with sharp gradients or complex boundaries, and knowing that a random or deterministic set of features can approximate them optimally gives scientists confidence in their numerical tools. The researchers also identified the precise scale at which the internal parameters of the network should be set to achieve these results, a detail that is crucial for practical implementation. By establishing that smooth sigmoidal activations preserve the full hierarchy of approximation rates, the study closes a long-standing gap in the mathematical understanding of neural networks. It confirms that these models are not just flexible curve-fitters but are theoretically sound instruments capable of capturing the full complexity of smooth, high-dimensional reality.