📊 statistics

Uncertainty propagation through trained multi-layer perceptrons: Exact analytical results

This paper presents exact analytical expressions for the mean and variance of the output of trained single-hidden-layer multi-layer perceptrons with ReLU activations when the input follows a multivariate Gaussian distribution, achieving this without relying on series expansions.

Original authors: Andrew Thompson, Miles McCrory

Published 2026-01-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Andrew Thompson, Miles McCrory

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but slightly mysterious, machine. You feed it some data (like a recipe), and it spits out a prediction (like a cake). This machine is a Multi-Layer Perceptron (MLP), a type of artificial intelligence.

The problem is: What if your recipe isn't perfect? Maybe you measured the flour with a shaky hand, or the temperature fluctuated. In the real world, your input data always has a little bit of "wobble" or uncertainty.

The big question this paper answers is: If we know exactly how "wobbly" our input is, can we calculate exactly how "wobbly" the final cake will be, without having to bake a million cakes to find out?

Here is the breakdown of their discovery using simple analogies:

1. The Machine: A Three-Step Factory

The authors looked at a specific, simplified version of these AI machines. Think of it as a factory with three stations:

Station 1 (The Mixer): You pour in your ingredients (the input data). The machine mixes them with some weights and adds a pinch of salt (this is a mathematical "affine transformation").
Station 2 (The Gatekeeper): This is the most important part. The machine uses a ReLU function. Imagine a gate that only lets positive numbers through. If the number is negative, the gate slams shut and the output becomes zero. It's like a "No Negative Numbers Allowed" rule.
Station 3 (The Baker): The numbers that made it through the gate are mixed again to create the final output (the prediction).

2. The Old Way: Guessing by Baking (Monte Carlo)

Before this paper, if you wanted to know how uncertain your final cake was, you had to use a method called Monte Carlo sampling.

The Analogy: Imagine you want to know how much the cake's height will vary if your flour measurement is off by a gram. The old way was to bake the cake 100,000 times, each time slightly changing the flour amount randomly. Then, you measure all 100,000 cakes, calculate the average height, and see how much they varied.
The Downside: It's slow, it takes a lot of computing power, and it's an estimate. You never get the exact answer, just a very good guess that gets better the more cakes you bake.

3. The New Way: The Exact Formula (Analytical Results)

The authors, Andrew Thompson and Miles McCrory, found a mathematical shortcut. They derived an exact formula (a closed-form expression) that tells you the exact average and the exact "wobble" (variance) of the output, just by looking at the math of the input.

The Analogy: Instead of baking 100,000 cakes, they wrote a single equation that tells you exactly how the "wobble" in the flour travels through the mixer, hits the gate, and changes the final height.
Why it's special:
- It's Exact: No guessing. No need to bake a million cakes.
- It's Transparent: You can look at the formula and see why the output is uncertain. It's like seeing the blueprint of the factory rather than just looking at the finished product.
- It's Fast: You don't need to run a simulation; you just plug the numbers into the formula.

4. The "Gatekeeper" Challenge

The hardest part of their math was the Gatekeeper (ReLU).

When you mix ingredients, the math is usually smooth and predictable (like a straight line).
But the Gatekeeper is tricky. It cuts off everything below zero. This creates a "kink" in the math.
Previous methods tried to approximate this kink using long, infinite series (like trying to describe a circle by adding more and more tiny straight lines).
The Breakthrough: The authors found a way to calculate the result of this "kink" using standard, well-known mathematical tools (Gaussian integrals) without needing those infinite, messy series. They solved the puzzle of how the "wobble" behaves when it hits the "No Negative Numbers" gate.

5. Did It Work? (The Test)

To prove their formula wasn't just theory, they tested it on a real-world problem: predicting the health of lithium-ion batteries (like in electric cars) using electrical data.

They trained their AI on battery data.
They took a set of test batteries and ran their Exact Formula to predict the uncertainty.
They also ran the Old Way (baking 1,000,000 virtual cakes via Monte Carlo sampling) to see what the "true" answer was.
The Result: The Exact Formula matched the 1,000,000-bake simulation almost perfectly, but in a fraction of the time.

Summary

This paper is like finding a perfect map for a journey through a complex, bumpy landscape.

Before: You had to walk the path a million times to figure out where the bumps were.
Now: You have a map that tells you exactly where the bumps are and how big they are, just by looking at the starting point.

They did this specifically for machines with one hidden layer and a ReLU gate. They admit that if the machine gets more complex (more layers) or uses different gates, the math gets much harder, but for this specific setup, they have the exact, perfect answer.

Technical Summary: Uncertainty Propagation through Trained Multi-Layer Perceptrons

Problem Statement
The paper addresses the challenge of uncertainty propagation through trained machine learning regression models, specifically Multi-Layer Perceptrons (MLPs). While uncertainty quantification is increasingly mandated for model trustworthiness (e.g., under the EU AI Act), existing methods for propagating known input uncertainties through a fixed model often rely on sampling-based approaches like Monte Carlo. These methods approximate the output distribution and require significant computational resources to achieve high accuracy. Analytical methods, which offer exact characterizations, are often limited to approximations (e.g., linearization, asymptotic limits) or rely on infinite series expansions that are computationally tedious to evaluate for specific activation functions.

Methodology
The authors focus on fully-connected feed-forward MLPs with a single hidden layer and ReLU activation functions, assuming the input data follows a multivariate Gaussian distribution. The architecture consists of:

An input vector $V$ passed through an affine transformation to produce $W$ .
A component-wise ReLU activation ( $X = W_+$ ) applied to $W$ .
A final affine transformation to produce the scalar output $Y$ .

The core methodological contribution is the derivation of exact analytical expressions for the mean and variance of the output $Y$ . The authors decompose the problem:

Propagation through the affine layers is handled via standard linear transformations of Gaussian distributions.
The primary challenge lies in propagating uncertainty through the ReLU activation. The authors derive closed-form expressions for the first and second moments (mean and covariance) of a rectified multivariate Gaussian distribution.
Specifically, they provide formulas for $E[X_i]$ , $E[X_i^2]$ , and $E[X_i X_j]$ (where $X = W_+$ ) using standard 1D and 2D Gaussian integrals involving the probability density function ( $\phi$ ) and cumulative distribution function ( $\Phi$ ).
The paper also addresses corner cases where input variances are zero or correlations are perfect ( $|\rho| = 1$ ), providing limiting forms for these scenarios.

Key Contributions

Exact Analytical Expressions: The paper presents closed-form formulas for the mean and variance of MLP outputs with ReLU activations and Gaussian inputs. Unlike previous work (e.g., [19]) that relies on infinite Taylor series, these expressions are finite and depend only on standard Gaussian integrals.
Handling of ReLU Non-linearity: The derivation explicitly handles the non-linearity of the ReLU function without resorting to linearization or mean-zero assumptions, providing exact results for the rectified distribution.
Corner Case Extensions: The authors extend their results to handle degenerate cases where input uncertainty is zero or where hidden layer variables are perfectly correlated.
Validation: The analytical results are validated against Monte Carlo sampling using a real-world test problem: estimating the State-of-Health (SOH) of lithium-ion cells based on Electrical Impedance Spectroscopy (EIS) data.

Results

Theoretical: The derived expressions (Theorems 1, 2, and 3) allow for the calculation of output moments to arbitrary precision without the need for series truncation.
Empirical: In the SOH prediction experiment, the analytical results were compared against Monte Carlo simulations with sample sizes ranging from $10^3$ $1 0^{3}$ to $10^6$ $1 0^{6}$ .
- The Root Mean Square Error (RMSE) between the analytical and Monte Carlo results decreased as the number of Monte Carlo trials increased.
- With $10^6$ trials, the RMSE for both mean and variance estimates dropped below $10^{-5}$ .
- Log-log plots of RMSE versus sample size confirmed a power-law relationship with a gradient of approximately $-0.5$, consistent with the expected $1/\sqrt{n}$ convergence rate of Monte Carlo methods. This confirms that the analytical expressions match the "true" values that Monte Carlo methods approximate.

Significance and Claims
The paper claims that its analytical approach offers a solution that is more accurate, transparent, and reproducible than sampling-based methods.

Accuracy: Analytical expressions precisely characterize output uncertainty, whereas sampling methods only approximate it.
Transparency: The mathematical derivation provides insight into the sources of propagated uncertainty, unlike "black box" sampling.
Reproducibility: Reproducing the evaluation requires only the input distribution and the analytical formula, avoiding the need to record or replicate specific sampling processes.

The authors note that while the current results are exact for single-hidden-layer networks with ReLU, extending these exact results to deeper networks or other activation functions would require different mathematical techniques or approximations (such as Gaussian approximations at each layer). The paper concludes that for the specific case of single-hidden-layer ReLU networks with Gaussian inputs, exact analytical uncertainty propagation is feasible and superior to sampling.