📊 statistics

Limitations of Learning Tanh Neural Networks with Finite Precision

This paper demonstrates that under finite-precision constraints, learning $\tanh$ neural networks containing localized bump functions is fundamentally limited to a Monte Carlo convergence rate unless the sampling budget grows exponentially with the network size, thereby extending known limitations from ReLU networks to the $\tanh$ setting.

Original authors: Philipp Grohs, Matěj Trödler

Published 2026-06-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Philipp Grohs, Matěj Trödler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to recognize a very specific, tiny secret hidden inside a vast, dark room. The computer is a "neural network," a type of AI that learns by looking at examples. In this paper, the authors are investigating a specific kind of AI that uses a mathematical tool called tanh (hyperbolic tangent) to process information. This tool is smooth and curved, unlike the "ReLU" tool used in many other AIs, which acts more like a sharp on/off switch.

The authors ask a fundamental question: How many "samples" (or peeks) does the computer need to take to perfectly learn this secret, assuming the computer has a limited ability to see very small numbers?

Here is the breakdown of their findings using simple analogies:

1. The "Fuzzy Vision" Problem (Finite Precision)

Imagine the computer is wearing glasses that are slightly foggy. If a number is smaller than a tiny speck of dust (let's call this the "machine precision"), the computer's glasses blur it completely, and it sees it as zero. It cannot tell the difference between a tiny whisper and total silence.

The authors show that because of this "fuzzy vision," the computer faces a massive hurdle. It can't distinguish between a function that is truly zero everywhere and a function that has a tiny, sharp "bump" hidden in a corner, unless that bump is huge enough to be seen through the fog.

2. The "Invisible Bump" Construction

The authors built a special mathematical trick to prove their point. They created a "bump" function (a small hill of data) that is:

Tall and sharp in the center (so it has a lot of "mass" or importance).
Exponentially thin on the edges.

Because the edges get thinner so fast, they eventually become so small that the computer's "fuzzy glasses" turn them into zero. To the computer, this bump looks exactly like a flat, empty floor everywhere except for one tiny, invisible spot.

3. The "Needle in a Haystack" Game

Now, imagine you are playing a game where you have to find these hidden bumps.

The Setup: You have a huge room (the data space). You can drop a limited number of "sensors" (samples) to check for bumps.
The Trap: The authors proved that if the bumps are hidden in a way that exploits the computer's "fuzzy vision," you can hide thousands of these bumps in the room.
The Result: Even if you drop a huge number of sensors, there is a high probability that none of your sensors will land on the tiny, hidden spots where the bumps actually exist. Your sensors will all read "zero" (because the bumps are invisible to them outside their tiny centers).

4. The "Exponential Cost"

This leads to the paper's main conclusion: Learning is incredibly expensive.

In the world of ReLU networks (the sharp on/off switches), the number of samples needed to learn grows somewhat predictably. But for these smooth tanh networks, the authors found that to guarantee you can learn the function accurately, the number of samples you need grows exponentially with the size of the network.

Think of it this way:

If you want to learn a small network, you might need 10 samples.
If you make the network slightly bigger, you might need 100 samples.
If you make it a bit bigger again, you might need 1,000,000 samples.
If you make it just a little bit bigger still, you might need more samples than there are atoms in the universe.

5. The "Unstable Truth"

The paper also highlights a scary instability. They showed that you could have two different functions that look identical to the computer (because the differences are smaller than the "fuzzy glasses" can see), but in reality, they are completely different (one has a big bump, the other doesn't).

Even if you had a perfect algorithm, the fact that the computer can't see the tiny differences means it can never be stable. A tiny, invisible change in the input could lead to a massive, unpredictable change in the output. It's like trying to balance a house of cards on a vibrating table; no matter how good your hands are, the table's vibration (the finite precision) makes a stable structure impossible.

Summary

The paper argues that for smooth, curved neural networks (tanh), finite precision acts as a hard wall. It prevents the computer from learning functions that have sharp, localized features unless you are willing to throw an astronomically large number of samples at the problem. In many realistic scenarios, this makes learning these specific types of networks computationally impossible, not because the math is too hard to solve, but because you simply don't have enough "eyes" (samples) to see the details before the computer's "foggy glasses" blur them away.

Technical Summary: Limitations of Learning Tanh Neural Networks with Finite Precision

Problem Statement
This paper investigates the fundamental limitations of learning classes of neural networks with hyperbolic tangent ( $\tanh$ ) activation functions under finite-precision arithmetic. While theoretical results establish that neural networks possess universal approximation capabilities, practical learning is constrained by the availability of only finitely many sampled values and the finite precision of computational devices. The authors address the question: How many samples $m$ are necessary to guarantee that an algorithm can approximate every function in a class of $\tanh$ networks to a specific accuracy $\epsilon$ ?

The study focuses on the interplay between the smooth, non-homogeneous nature of $\tanh$ activations and the constraints imposed by finite precision (quantization). Unlike ReLU networks, where localized "bump" functions can be constructed exactly due to the homogeneity property ( $\rho(\lambda x) = \lambda \rho(x)$ ), $\tanh$ networks are real-analytic and generally cannot produce functions with compact support (exact zero on sets of positive measure). This creates a significant analytical hurdle for proving lower bounds on learning complexity, as standard techniques relying on exact localization fail.

Methodology
The authors develop a novel framework to overcome the lack of exact compact support in $\tanh$ networks by leveraging finite-precision arithmetic. Their approach involves three primary technical components:

Finite-Precision Localization: The authors assume a quantization map $Q_{\epsilon_p}$ where any value with magnitude $|x| \le \epsilon_p$ is rounded to zero. They construct neural networks that produce "sharply localized bumps": functions that are non-trivial on a small cube but decay exponentially fast outside this region. By tuning the network depth and parameters, the "tails" of these functions are forced to fall below the machine precision threshold $\epsilon_p$ , rendering them indistinguishable from zero by the computational device, despite having positive $L_p$ norm.
Fixed-Point Iteration Mechanism: A core innovation is the exploitation of the fixed-point structure of the $\tanh$ activation. For a scaled activation $\sigma_a(x) = \tanh(ax)$ with $a > 1$ , there exist non-trivial fixed points $\pm x_\star$ satisfying $\tanh(ax_\star) = x_\star$ . The authors design network layers that iteratively apply $\sigma_a$ to a coarse bump function. This iteration amplifies the signal, driving the output toward a step-function profile (converging to $\pm x_\star$ inside the support and decaying rapidly outside). This mechanism replaces the homogeneity-based constructions used in ReLU analysis.
Adaptive Randomized Algorithm Framework: The lower bounds are derived for adaptive randomized algorithms, which can choose sampling points based on previous observations and use internal randomness. The proof utilizes a "pigeonhole" argument: by constructing a large family of disjoint bump functions (centered on a grid) and showing that their supports are smaller than the sampling density, the authors demonstrate that a significant fraction of these functions will yield zero quantized values at all sampled points. Consequently, the algorithm cannot distinguish these functions from the zero function.

Key Contributions and Results
The paper establishes a lower bound on the sampling complexity for learning $\tanh$ neural networks. The main result (Theorem 6) proves that for a class of networks with input dimension $d$ , depth $L$ , width $B$ , and bounded coefficients, the approximation error $\epsilon$ for any algorithm using $m$ samples satisfies:
$\epsilon \gtrsim C \cdot m^{-1/p}$
where the constant $C$ depends on the network architecture parameters.

Key findings include:

Exponential Dependence on Architecture: The admissible number of samples $m$ required to achieve a fixed accuracy grows exponentially with respect to the network architecture parameters (specifically width and depth). This implies that learning becomes infeasible as the network size increases, even if the optimization problem were computationally solvable.
Comparison with ReLU: While the ReLU setting (Berner et al., 2023) exhibits a lower bound scaling with $m^{-1/d - 1/p}$ , the $\tanh$ setting under finite precision yields a bound scaling with $m^{-1/p}$ . However, the constant factor in the $\tanh$ bound depends exponentially on the architecture, leading to a similar practical infeasibility.
Intrinsic Instability: The results reveal a fundamental instability in the learning problem. For any algorithm using a sub-exponential number of samples, there exist pairs of networks that differ by a negligible perturbation (below machine precision) at all sampled points but have a large $L_p$ distance (order 0.5). Thus, no reconstruction procedure can be simultaneously accurate and stable within the stated sampling regime.
Role of Precision: The hardness result is contingent on the finite-precision assumption. The construction relies on the ability to "hide" the tail of the bump function below the quantization threshold $\epsilon_p$ .

Significance and Claims
The paper claims to extend previous hardness results for ReLU networks to the widely used $\tanh$ activation, which is prevalent in recurrent architectures (RNNs, LSTMs), physics-informed neural networks (PINNs), and modern foundational models.

The authors emphasize that their results are not merely about computational complexity (running time) but about information-theoretic limitations. Even if an optimization algorithm could find the global minimum, the number of samples required to uniquely identify the function from point evaluations is prohibitively large. The work highlights that the smoothness and analytic nature of $\tanh$ , often viewed as advantageous for differentiability, introduce severe challenges for learnability in finite-precision environments, particularly regarding the construction of localized features necessary for proving lower bounds.

The paper provides a simplified theorem (Theorem 1) and a full technical statement (Theorem 6) with explicit dependencies on parameters such as machine precision $\epsilon_p$ , network width $B$ , depth $L$ , and coefficient bounds $c$ . It includes a numerical verification tool to demonstrate that for moderate network sizes and realistic machine precisions, the required sample sizes for uniform recovery become astronomically large, rendering the learning task computationally infeasible.