🤖 machine learning

Expressive Power of Floating-Point Neural Networks with Arbitrary Reduction Orders and Inexact Activation Implementations

This paper establishes a general distinguishability framework to characterize the universal representability of floating-point neural networks under realistic execution semantics, proving that arbitrary reduction orders and bounded ulp errors in activation implementations do not preclude exact function representation for a broad class of practical activation functions.

Original authors: Yeachan Park, Geonho Hwang, Wonyeol Lee, Sejun Park

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yeachan Park, Geonho Hwang, Wonyeol Lee, Sejun Park

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a neural network as a giant, complex factory designed to sort and transform raw materials (data) into finished products (answers). For decades, the blueprints for these factories were drawn by mathematicians who assumed the factory had perfect tools. They assumed that if you added two numbers, the result was always exactly correct, and that the factory's "activation switches" (the parts that decide how much signal to pass along) worked with absolute, mathematical precision.

However, real-world computers don't use perfect tools. They use floating-point arithmetic, which is like a factory using rulers that are slightly worn down. When you add numbers, the order in which you add them can change the result (because the ruler isn't perfect), and the "activation switches" might not be set to the exact theoretical position—they might be off by a tiny fraction of a hair's width.

This paper asks a critical question: If we build our neural network factory with these imperfect, real-world tools, can it still do everything we need it to do? Specifically, can it still learn to represent any possible pattern or function, or does the imperfection break the magic?

Here is the breakdown of their findings using simple analogies:

1. The "Order of Operations" Problem

In a perfect world, adding numbers is like stacking blocks: it doesn't matter if you stack block A on B, then C, or B on C, then A; the tower is the same.
In the real world (floating-point), the order matters. It's like trying to mix paint in a bucket where the bucket is slightly leaky. If you pour the red paint in first, then the blue, you get a slightly different shade than if you pour blue first, then red.

The Paper's Finding: The authors proved that even if the factory uses any random order to mix these paints (add numbers), the network can still learn anything, provided the "activation switches" are good enough. You don't need a fixed, perfect order to get the job done.

2. The "Distinguishability" Test

To understand how a factory sorts items, imagine you have two very similar-looking apples (Input A and Input B).

The Problem: If the factory's first machine (the first layer) squashes both apples into the exact same shape, the rest of the factory will never know they were different. It will treat them as the same apple forever.
The Paper's Solution: The authors introduced a rule called "Distinguishability." They proved that for a network to be a "universal" learner (able to do anything), its first layer must be able to tell every single pair of different inputs apart. If the first layer can't tell two different inputs apart, the whole network fails.
The Good News: They showed that most common activation functions (like ReLU, Sigmoid, Tanh, Swish, etc.) can tell inputs apart, even with imperfect math.

3. The "Imperfect Switch" Problem

In theory, a switch might turn on exactly when the input hits 0.5. In reality, due to manufacturing limits, the switch might turn on at 0.5000001 or 0.4999999.

The Paper's Finding: Previous theories said, "If the switch isn't perfect, the network might break." This paper says, "Not necessarily."
They proved that as long as the "imperfection" (the error) is small and bounded (like being off by only a few tiny units, or "ulps"), the network can still distinguish inputs and learn anything.
The Result: They confirmed that popular activation functions used in real life—like Sigmoid, Tanh, ReLU, GELU, Swish, and even Sin—are robust enough to work perfectly well, even if their computer code implementations aren't mathematically perfect.

4. The "Cosine" Counter-Example

The authors also found a specific case where the factory does break. They showed that if you use a Cosine activation function (which wiggles up and down like a wave), the network might fail to distinguish between certain inputs because the wave repeats itself. It's like having a factory where two different colored balls get painted the exact same shade of blue because the paint machine cycles through colors too fast.

This explains why some theoretical functions work in math class but fail in real computer code.

Summary: The Big Takeaway

The paper essentially says: "Don't worry about the tiny errors in your computer's math."

Even though real computers have:

Rulers that aren't perfect (rounding errors),
Mixing orders that change the result (non-associative addition),
Switches that aren't perfectly calibrated (inexact activation implementations),

...neural networks built with these tools are still powerful enough to represent any function you throw at them, as long as you use standard activation functions (like ReLU or Sigmoid). The "magic" of neural networks survives the transition from perfect math theory to messy, real-world engineering.

Technical Summary: Expressive Power of Floating-Point Neural Networks with Arbitrary Reduction Orders and Inexact Activation Implementations

1. Problem Statement

Existing theoretical frameworks for the expressive power of neural networks predominantly rely on exact real arithmetic. However, practical neural networks operate under finite-precision floating-point arithmetic, characterized by:

Finite Representability: Parameters and activations belong to a discrete set of machine numbers.
Rounding Errors: Arithmetic operations incur rounding errors.
Non-Associativity: Floating-point addition is non-associative, meaning the output of affine transformations depends on the reduction order (the sequence in which terms are summed).
Inexact Implementations: Practical implementations of activation functions (e.g., exp, tanh, sin) often deviate from correctly rounded values by several units in the last place (ulp) to optimize computational cost.

Previous works addressing floating-point expressivity have been limited by idealized assumptions, such as:

Restricting analysis to piecewise-linear activations (e.g., ReLU, Step).
Assuming fixed left-to-right reduction orders.
Assuming correctly rounded activation implementations.

These assumptions fail to capture the behavior of networks in modern, massively parallel environments (e.g., GPUs) where reduction orders vary, and where smooth, widely used activations (e.g., Sigmoid, GELU, Swish) are implemented with bounded approximation errors. Consequently, it remains unclear whether floating-point neural networks retain their universal representability under these realistic conditions.

2. Methodology

The authors introduce a general framework to analyze the universal representability of floating-point neural networks. A network is considered a "universal representator" if it can represent every function between prescribed floating-point domains exactly.

Key Definitions and Framework

Floating-Point Neural Networks ( $N(I,S)$ ): Defined under an arbitrary reduction order $S$ . The affine transformation in layer $\ell$ computes $\Sigma((w_i \otimes x); o_{d_{in}}) \oplus b_i$ , where $\Sigma$ denotes the reduction sum according to order $o$ , and $\otimes, \oplus$ are floating-point multiplication and addition.
Distinguishability: The core concept introduced is $\sigma$ -distinguishability. A domain $X$ $X$ is $\sigma$ $σ$ -distinguishable under $S$ $S$ if for any distinct $x, x' \in X$ $x, x^{'} \in X$ , there exists a floating-point affine transformation $\phi$ $ϕ$ such that $\sigma(\phi(x)) \neq \sigma(\phi(x'))$ $σ (ϕ (x)) \neq = σ (ϕ (x^{'}))$ .
- Necessity: The paper proves that if a domain is not $\sigma$ -distinguishable, the network cannot represent all functions on that domain (Lemma 2).
- Sufficiency: Under mild conditions on the activation function (Condition 1), $\sigma$ -distinguishability is shown to be sufficient for universal representability (Theorem 5).

Technical Approach

Separating Points: The authors define "separating points" ( $\eta$ ) where the activation function's output changes discontinuously relative to the floating-point grid (i.e., $\sigma(\eta^-) \notin \{\sigma(\eta), \sigma(\eta^+)\}$ ). They establish conditions under which these points exist for both correctly rounded and inexact implementations.
Transferability: A series of technical lemmas (Section V) demonstrate that under Condition 1, specific pairs of floating-point numbers can be "transferred" to target values using left-to-right additions and activation functions. This allows the construction of networks that isolate specific inputs.
Handling Inexactness: For implementations with bounded ulp errors, the authors derive conditions (Condition 2) ensuring that the approximation error does not collapse the necessary separating points, provided the precision $p$ is sufficiently high (e.g., $p \geq 7$ ).

3. Key Contributions

A. General Distinguishability Framework

The paper establishes a necessary and sufficient condition for universal representability based on distinguishability.

Necessary Condition: If a network cannot distinguish every pair of distinct inputs in the first layer via an affine transformation followed by activation, it cannot represent all functions. This framework generalizes previous counterexamples (e.g., correctly rounded cosine) to broader classes of functions.
Sufficient Condition: If the activation function satisfies Condition 1 (existence of specific output values $C_0, C_1, C_2$ ) and the domain is $\sigma$ -distinguishable, a four-layer network can represent any function on that domain. Crucially, this holds for arbitrary reduction orders, proving that universal representability is not an artifact of a specific summation sequence.

B. Extension to Practical Activation Functions

The theory extends beyond piecewise-linear functions to a broad class of smooth, practical activations:

Correctly Rounded: The paper proves universal representability for correctly rounded versions of Sigmoid, tanh, ReLU, ELU, SeLU, GELU, Swish, Mish, and sin.
Inexact Implementations: The authors provide verifiable conditions under which implementations with bounded ulp errors (up to $K=10$ ulps) remain universal representators. This covers practical libraries where functions like exp and sin are approximated.

C. Negative Results and Counterexamples

The framework identifies specific activation functions that fail to be universal representators under certain conditions.

Corollary 4: Networks using correctly rounded cosine, $\sqrt{1+x^2}$ , cosh, or $e^{-x^2}$ cannot represent all functions on specific domains (e.g., $[-2^{p+3}, 2^{p+3}]_F$ ) because they fail the distinguishability condition. This generalizes the known counterexample of the cosine activation.

4. Results

The paper establishes the following main theorems:

Theorem 5: If $\sigma$ satisfies Condition 1 and $X$ is $\sigma$ -distinguishable with range $[-2^{e_{max}}, 2^{e_{max}}]_F$ , then any function $f: X \to (F \cup \{-\infty, \infty\})^{d_{out}}$ can be represented by a four-layer $\sigma$ network under any reduction order $S$ .
Corollary 9 & 12: For correctly rounded Sigmoid, tanh, ReLU, ELU, SeLU, GELU, Swish, Mish, and sin, universal representability holds over large domains (e.g., $(-2^{e_{max}-1}, 2^{e_{max}-1})_F^d$ ). Notably, even the Identity activation function is shown to be a universal representator in floating-point arithmetic (unlike in real arithmetic) due to the non-affine nature of floating-point operations.
Corollary 13 & 16: For Sigmoid, tanh, ReLU, ELU, SeLU, GELU, Swish, Mish, and sin implemented with bounded ulp errors (Condition 2), universal representability holds provided the precision $p \geq 7$ .

5. Significance and Claims

The paper claims to substantially generalize the theoretical understanding of floating-point neural networks:

Realism: It moves beyond idealized assumptions (fixed reduction orders, correct rounding) to models that reflect actual hardware execution (arbitrary orders, bounded ulp errors).
Broad Applicability: It provides the first theoretical guarantees for the expressive power of networks using smooth, non-piecewise-linear activations (like GELU and Swish) under realistic floating-point semantics.
Fundamental Insight: It demonstrates that the "universal approximation" phenomenon in floating-point arithmetic is robust to the non-associativity of addition and implementation inaccuracies, provided the activation function maintains sufficient "separating" properties.
Limitations: The results are modest regarding low-precision formats (e.g., float8), where the number of representable values is too small for the derived conditions to hold without specific lookup table implementations. The paper also notes that the proofs rely on the existence of specific separating points, which may fail for certain functions (like cosine) or very low precisions.

The authors emphasize that these results are derived from a new framework that does not rely on piecewise-linearity, fixed summation orders, or exact activation behaviors, thereby offering a more rigorous foundation for understanding neural networks in practical, finite-precision settings.