🤖 machine learning

Floating-Point Networks with Automatic Differentiation Can Represent Almost All Floating-Point Functions and Their Gradients

This paper demonstrates that neural networks operating under practical floating-point arithmetic and automatic differentiation can theoretically represent arbitrary target function values and their corresponding gradients, extending universal approximation results to real-world computational constraints.

Original authors: Sejun Park, Yeachan Park, Geonho Hwang

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sejun Park, Yeachan Park, Geonho Hwang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to solve a puzzle. In the world of pure mathematics, we assume the robot has an infinite supply of perfect, infinitely precise numbers (like real numbers) and can perform calculations without ever making a tiny mistake. Under these perfect conditions, we already knew that a neural network (the robot's brain) could learn not just the answer to the puzzle, but also how the answer changes if you tweak the puzzle slightly. This "how it changes" is called a gradient.

However, real computers don't work with perfect numbers. They use floating-point numbers, which are like a limited set of stepping stones across a river. Because there are only so many stones, the computer has to round off numbers, leading to tiny errors (round-off errors). Furthermore, the way computers calculate these changes (using a method called Automatic Differentiation) is a specific, step-by-step recipe that depends on these tiny errors.

This paper asks a big question: Can a real-world computer brain, with all its rounding errors and limited stepping stones, still learn to produce any desired answer and any desired "change" (gradient) at the same time?

The Core Discovery: The "Magic Trick" of Rounding

The authors say yes, and they prove it with a clever trick.

Think of a neural network as a factory assembly line.

The Old View: In the perfect math world, if you want the factory to output a specific number and a specific "rate of change," you just tune the machines.
The Real World Problem: In the real computer world, the "machines" (math operations) have a glitch: they aren't perfectly consistent. If you multiply three numbers in a different order, you might get a slightly different result because of rounding. This is called non-associativity.

The authors discovered that this "glitch" is actually a superpower.

They show that you can build a floating-point network that acts like a two-faced coin:

Face A (The Output): It produces the exact answer you want (e.g., "The temperature is 25 degrees").
Face B (The Gradient): It produces any change signal you want (e.g., "If you change the input by a tiny bit, the output changes by exactly this much"), even if that change signal has nothing to do with the math of the answer itself.

The "Two-Track" Analogy

Imagine you are building a machine that takes an input and gives you a result.

Track 1 (The Answer): You want the machine to say "Hello."
Track 2 (The Reaction): You want the machine to scream "Fire!" if you nudge it, even though "Hello" and "Fire!" have no logical connection.

In a perfect math world, the reaction ("Fire!") would have to be mathematically tied to the answer ("Hello"). If you change the answer, the reaction changes proportionally. You can't just pick them independently.

But in the floating-point world, the authors show you can use the "rounding errors" as a secret code. By carefully arranging the steps of the calculation (like arranging the order of multiplication), the machine can:

Calculate the answer perfectly.
Simultaneously calculate a "reaction" that is completely independent of the answer, effectively hacking the gradient.

Why This Matters (According to the Paper)

The paper doesn't talk about curing diseases or building self-driving cars. Instead, it focuses on the theoretical limits of what these networks can do:

Total Control: You can make a network fit any set of data points and any set of gradients you want, as long as you have enough layers (depth) in your network. It's like having a universal remote that can make the TV display any picture and react to any button press in any way you choose.
Security and Privacy: Because you can manipulate the gradients independently, you could theoretically train a network to give the right answer but hide the "clues" (gradients) that attackers use to reverse-engineer the data. You can make the network say "Yes" while its "whispers" (gradients) say "Nothing to see here."
Breaking the Rules of Math: The paper highlights a fundamental difference between "perfect math" and "computer math." In perfect math, the gradient is a slave to the function. In computer math, thanks to rounding errors, the gradient can be a free agent.

The "Ingredients"

The authors proved this works for the most common "activation functions" (the switches inside the brain) used today, such as:

ReLU (The most common switch)
Sigmoid and Tanh (S-shaped curves)
Swish, GELU, ELU (Newer, smoother switches)

They showed that as long as the computer uses standard formats (like 16-bit, 32-bit, or 64-bit floats), this "magic trick" works.

Summary

In simple terms: Floating-point neural networks are more flexible than we thought. Because computers make tiny rounding mistakes, they can actually be programmed to output any answer and any "change signal" simultaneously, even if those two things shouldn't logically go together. The paper proves that these networks are powerful enough to represent almost any function and its gradient, turning a computer's limitation (rounding errors) into a feature for total control.

Technical Summary: Floating-Point Networks with Automatic Differentiation Can Represent Almost All Floating-Point Functions and Their Gradients

1. Problem Statement

Modern neural network applications frequently require not only the approximation of function values but also the computation of gradients with respect to inputs. These gradients are critical for tasks such as scientific machine learning (solving partial differential equations), sensitivity analysis, inverse problems, reinforcement learning, adversarial attacks, and input generation.

Theoretical results in approximation theory have established that neural networks with real-valued parameters and exact arithmetic can approximate both differentiable functions and their gradients on compact domains. However, these theoretical guarantees do not directly apply to practical implementations. Real-world neural networks operate using floating-point arithmetic (IEEE 754 standard), where parameters and intermediate values are restricted to a finite set of numbers, and operations involve rounding errors. Consequently, a practical network is an algorithm defined by a sequence of floating-point operations rather than a continuous real-valued function. Furthermore, gradients in these systems are computed via Automatic Differentiation (AD) (e.g., backpropagation) applied to this computational graph. Under floating-point arithmetic, AD outputs are not necessarily the derivatives of a differentiable real function but are instead the results of a specific computational procedure subject to non-associativity and rounding.

The central problem addressed in this work is: Can floating-point neural networks, when paired with automatic differentiation, represent arbitrary function values and arbitrary gradients?

2. Methodology and Framework

2.1 Definitions and Setup

The authors define a floating-point neural network $f$ as a composition of floating-point affine transformations $\rho_l$ and a pointwise activation function $\sigma$ :
$f(x) = \rho_L \circ \sigma \circ \rho_{L-1} \circ \dots \circ \sigma \circ \rho_1(x)$
where $\rho_l(x) = A_l \otimes x \oplus b_l$ . Here, $\otimes$ and $\oplus$ denote floating-point multiplication and addition, respectively, which include rounding operations.

The gradient computation is modeled using the reverse-mode automatic differentiation algorithm, denoted as $DAD$. For a composite function $\phi_x \circ f$ , where $\phi_x$ is a scalar function (e.g., a loss function) and $\phi'_x$ is its derivative, the gradient is computed as:
$DAD(\phi_x \circ f, x) = \phi'_x(f(x)) \otimes \rho'_L(z_L) \otimes \sigma'(y_{L-1}) \otimes \dots \otimes \rho'_1(x)$
Crucially, the multiplications are evaluated in a left-associative manner ( $((a \otimes b) \otimes c) \dots$ ) to account for the non-associativity of floating-point operations. This non-associativity is a key mechanism exploited in the proofs.

2.2 Network Construction Strategy

To prove representability, the authors construct specific networks to decouple function value representation from gradient representation. The core strategy involves combining two sub-networks, $f_1$ and $f_2$ , using a concatenation operation ( $\#$ ):

$f_1$ (Function Value Network): Designed to output the target function values $f^*(x)$ while producing a zero gradient ( $D f_1, x = 0$ ). This is achieved by constructing indicator functions for specific inputs and suppressing gradients through sequential multiplication by small weights.
$f_2$ (Gradient Network): Designed to output zero function values ( $f_2(x) = 0$ ) while producing a specific target gradient $g^*(x)$ when the input gradient is non-zero. This network utilizes the non-associativity of floating-point multiplication to manipulate the gradient magnitude and direction independently of the function output.

The final network is constructed as $f = f_2 \# f_1$ , allowing the network to simultaneously satisfy $f(x) = f^*(x)$ and $DAD(\phi_x \circ f, x) = g^*(x)$ .

2.3 Conditions on Activation Functions

The proofs rely on specific technical conditions (Conditions 1–3) imposed on the activation function $\sigma$ and its derivative $\sigma'$ . These conditions ensure the ability to:

Distinguish inputs (Condition 1).
Suppress or amplify gradients without overflow (Condition 2).
Generate specific gradient values in the first layer (Condition 3).

The authors verify that these conditions hold for a wide range of practical activation functions, including ReLU, ELU, GELU, Swish, Sigmoid, and tanh, under standard floating-point formats (float16, float32, float64, bfloat16, and 8-bit formats).

3. Key Contributions and Results

3.1 Representation of Arbitrary Values and Gradients (Theorem 3.1 / Theorem 1)

The primary result establishes that for almost all target functions $f^*$ and target gradients $g^*$ (with a mild consistency condition where $g^*(x)=0$ if the input gradient is zero), there exists a floating-point network $f$ such that:
$f(x) = f^*(x) \quad \text{and} \quad DAD(\phi_x \circ f, x) = g^*(x)$
This holds for networks with as few as 7 layers for standard activation functions (ReLU, ELU, GELU, Swish) and slightly more for Sigmoid and tanh, provided the domain is within a specific range (e.g., $[-\Omega/8, \Omega/8]^d$ ).

3.2 Representation of Gradients Dependent on Input Derivatives (Theorem 3.2 / Theorem 2)

The authors extend the result to show that a single network $f$ can represent gradients that are arbitrary functions of both the input $x$ and the input gradient $y = \phi'_x(f(x))$ . Specifically, for almost all $f^*$ and $g^*(x, y)$ satisfying the symmetry $g^*(x, -y) = -g^*(x, y)$ , there exists a network $f$ such that:
$f(x) = f^*(x) \quad \text{and} \quad DAD(\phi_x \circ f, x) = g^*(x, \phi'_x(f(x)))$
This result highlights a fundamental discrepancy between exact and floating-point arithmetic. In exact arithmetic, the gradient of a composition $\nabla(\phi \circ f)$ must be proportional to $\phi'(f(x))$ . In floating-point arithmetic, due to non-associativity and rounding, the network can produce gradients that are not proportional to the input gradient, effectively "manipulating" the gradient while maintaining the correct function output.

3.3 Generalization to Other Activation Functions

The paper provides generalized theorems (Theorems 5 and 7) for activation functions satisfying Conditions 1–3, extending the results beyond the specific set $\Sigma$ to general activation functions.

4. Significance and Claims

The paper claims to demonstrate that floating-point neural networks with automatic differentiation possess universal representational power for both function values and gradients, a property not guaranteed by classical approximation theory which assumes real arithmetic.

The significance of this work lies in:

Bridging Theory and Practice: It validates that the theoretical capability of neural networks to fit gradients extends to the actual algorithms used in modern deep learning frameworks (PyTorch, TensorFlow), which rely on floating-point arithmetic.
Gradient Manipulation: It reveals that floating-point networks can perform target tasks (fitting function values) while simultaneously having "manipulated" gradients. This implies that gradients computed via AD in practical settings may not strictly reflect the derivative of the underlying mathematical function but are instead artifacts of the computational graph and rounding.
Security and Privacy Implications: The ability to fit function values while outputting arbitrary (e.g., zero or misleading) gradients suggests potential mechanisms for protecting against gradient-based attacks or reasoning, as the gradient information can be decoupled from the function behavior.
Scientific Machine Learning: The results support the theoretical foundation for using neural networks as surrogates for physical models where both the solution and its sensitivity (gradient) are required, confirming that finite-precision implementations can theoretically achieve the necessary fidelity.

The authors maintain a modest tone, noting that their results hold for "almost all" functions (excluding a measure-zero set of pathological cases) and rely on specific conditions regarding the domain size and floating-point precision (e.g., requiring $E \ge 6$ for the main theorems, though extensions exist for lower precision). They do not claim that all possible gradient behaviors are achievable without constraints (e.g., the symmetry constraint $g^*(x, -y) = -g^*(x, y)$ is inherent to the definition of AD).