📊 statistics

Uncertainty propagation in auto-regressive random neural network models

This paper presents analytical and particle-based methods for propagating uncertainty in random neural networks with Leaky ReLU activations, deriving closed-form approximations for output statistics and recursive equations for state-parameter cross-covariances in autoregressive dynamical systems, which are validated on high-dimensional models like Lorenz-63 and Kuramoto-Sivashinsky.

Original authors: Janice Adams, Daniele Venturi

Published 2026-08-24

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Janice Adams, Daniele Venturi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world of science and engineering, computers have become indispensable tools for predicting how complex systems will behave. From forecasting the weather to modeling the flow of blood through an artery, these digital models rely on artificial neural networks, which are mathematical structures inspired by the human brain. These networks learn to recognize patterns and make predictions by adjusting millions of internal settings, known as parameters, based on vast amounts of data. However, a significant challenge remains: these models are rarely perfect. The data fed into them often contains small errors, and the internal settings themselves are never known with absolute certainty. In many real-world scenarios, these tiny uncertainties can be amplified as the model runs, leading to predictions that drift wildly from reality or become completely unstable. Scientists have long sought ways to track how these uncertainties grow over time, but traditional methods often struggle when the system being modeled is chaotic and changes rapidly.

A team of researchers at the University of California, Santa Cruz, has developed a new way to track these uncertainties through neural networks, specifically for systems that predict their own future steps. Their approach treats both the input data and the internal settings of the network as variables that can fluctuate, rather than fixed numbers. By leveraging a specific type of mathematical function used in these networks, they created a method that can analytically calculate how uncertainty spreads without needing to run thousands of separate simulations. They tested this method on two notoriously difficult systems: the Lorenz-63 model, a classic example of chaotic weather-like behavior, and the Kuramoto–Sivashinsky equation, which describes complex fluid dynamics. The results show that their technique can accurately follow the growth of uncertainty for as long as the system remains predictable, offering a more efficient and stable alternative to existing methods.

The core of the problem lies in how neural networks handle uncertainty. When a network makes a prediction, it takes an input, processes it through layers of mathematical operations, and produces an output. If the input is slightly off, or if the internal settings are slightly different from what they should be, the output changes. In a simple system, this change might be small and manageable. But in a chaotic system, where small differences lead to vastly different outcomes, these errors can explode. To understand this, imagine trying to predict the path of a leaf floating down a turbulent river. If you are slightly off about the leaf's starting position, your prediction of where it will be in a minute might be completely wrong. The researchers focused on a specific type of neural network where the internal math is built using a function called Leaky ReLU. This function has a unique property: it is made of straight lines. While the overall network is complex, this piecewise linear structure allows the researchers to create a precise local map of how small changes in the input or settings affect the output.

Using this insight, the team derived a set of equations that describe how the average prediction and the spread of possible outcomes evolve over time. Instead of guessing, they calculated the exact mathematical relationship between the uncertainty in the inputs, the uncertainty in the network's settings, and the resulting uncertainty in the prediction. A crucial part of their discovery was realizing that as the network runs step-by-step, the uncertainty in the state of the system and the uncertainty in the network's settings become linked. They developed a way to track this connection, which they call a cross-covariance, ensuring that the model accounts for how the two types of uncertainty influence each other. This is vital because ignoring this link leads to inaccurate predictions.

The researchers first tested their method on the Lorenz-63 system, a mathematical model that mimics the chaotic behavior of atmospheric convection. In this system, the predictability horizon—the time limit beyond which predictions become useless—is roughly 5.11 time units. The team compared their new method against a standard approach known as Monte Carlo simulation, which involves running the model thousands of times with slightly different inputs to see the range of outcomes. While the standard method is accurate, it is computationally expensive. The new method, which they call Resampled Moment Propagation, achieved similar accuracy but with far fewer computational resources. In their tests, using just 100 particles (representing different possible scenarios) was enough to match the results of 3,000 samples from the standard method. The new approach successfully tracked the mean prediction and the spread of uncertainty all the way to the edge of predictability, whereas a simpler version of their method that did not include resampling became unstable and produced wildly incorrect results after just a short time.

Encouraged by these results, the team applied their method to a much more complex system: the Kuramoto–Sivashinsky equation. This model describes the behavior of a thin film of fluid flowing down a surface, a process that is highly chaotic and involves hundreds of variables. The researchers discretized the system into 200 dimensions, creating a massive computational challenge. Here, the predictability horizon was longer, extending to about 41.84 time units. The team ran their simulation for 500 time steps, which covered a significant portion of this horizon. Even in this high-dimensional environment, the method held up well. It accurately captured the growth of uncertainty and the average behavior of the system. However, the researchers noted a practical limitation: the method requires a large amount of computer memory to store the connections between the system's state and the network's parameters. For the 200-dimensional system, this required about 48 gigabytes of storage just for these connections. Despite this memory demand, the method proved that it is possible to propagate uncertainty through high-dimensional, chaotic systems without resorting to the brute force of running thousands of full simulations.

The success of this approach hinges on a technique called resampling. In their method, the researchers track a collection of possible scenarios, or particles. As time passes, the uncertainty associated with each particle grows. If this growth is allowed to continue unchecked, the model becomes unstable. To prevent this, the researchers periodically "reset" the system. At specific intervals, they take the current state of each particle, which is represented by a local cloud of possibilities, and draw a new sample from it. This process refreshes the collection of particles, preventing the uncertainty from spiraling out of control while still allowing the model to capture the complex, non-linear ways in which uncertainty evolves. They found that the timing of these resets is critical. If they happen too often, the model fails to accumulate enough uncertainty and underestimates the risk. If they happen too rarely, the model becomes unstable. Through their experiments, they determined that a specific interval, such as every 10 to 20 time steps, provided the best balance for the systems they tested.

This work represents a significant step forward in making artificial intelligence more reliable for scientific applications. By providing a way to mathematically track uncertainty through complex, chaotic systems, the researchers have offered a tool that can help scientists understand the limits of their predictions. The method does not eliminate uncertainty, but it provides a clear, accurate picture of how that uncertainty grows and evolves. This is particularly important for fields like climate modeling and fluid dynamics, where small errors can lead to large-scale misinterpretations. The researchers acknowledge that their method is currently limited by the memory required for very large systems, but they suggest that future improvements could reduce this cost. Their findings demonstrate that by combining the structural properties of neural networks with careful mathematical analysis, it is possible to build models that are not only predictive but also honest about their own limitations. The ability to forecast how far ahead a prediction can be trusted is just as valuable as the prediction itself, and this new framework provides a robust way to determine that boundary.

Technical Summary: Uncertainty Propagation in Auto-Regressive Random Neural Network Models

Problem Statement
Scientific Machine Learning (SciML) has advanced rapidly, yet reliable methods for assessing the uncertainty and trustworthiness of neural network predictions remain underdeveloped, particularly for auto-regressive applications involving nonlinear dynamical systems. In these settings, small uncertainties in inputs or model parameters can be amplified by the network, leading to inaccurate or unstable predictions. While Bayesian approaches (e.g., Variational Inference, Monte Carlo Dropout) exist for characterizing model uncertainty, their application to auto-regressive prediction settings—where the network output at one time step becomes the uncertain input for the next—remains largely unexplored. Standard Monte Carlo methods are computationally expensive, and single-Gaussian propagation often becomes unstable over long time horizons due to the accumulation of non-Gaussian features and unbounded variance growth.

Methodology
The authors develop a framework for uncertainty propagation in random neural networks where both inputs ( $x$ ) and network parameters ( $\theta$ ) are treated as random variables. The methodology relies on the piecewise-linear structure of the Leaky ReLU activation function.

Local Linearization: The authors derive a first-order Taylor expansion of the network output $F_\theta(x)$ with respect to both inputs and parameters. Because Leaky ReLU is piecewise linear, this approximation is exact within regions where the activation pattern (the set of active neurons) remains unchanged. The approximation is given by:
$F_{\theta+\Delta\theta}(x+\Delta x) \approx F_{\bar{\theta}}(\bar{x}) + J_x(x, \theta)\Delta x + J_\theta(x, \theta)\Delta \theta$
where $J_x$ and $J_\theta$ are the Jacobians with respect to inputs and parameters, respectively.
Analytical Propagation: Using this linearization, the authors derive analytical expressions for the Probability Density Function (PDF) and characteristic function of the network output. For Gaussian perturbations, the output distribution is shown to be Gaussian with a mean and covariance computable in closed form via the combined Jacobian. The framework explicitly accounts for the cross-covariance between state and parameter perturbations, which is critical as these become statistically dependent over time in auto-regressive settings.
Auto-Regressive Extension: The framework is extended to autonomous dynamical systems where the evolution map is a random neural network ( $x_{t+1} = F_\theta(x_t)$ ). The authors derive recursive equations for the mean, covariance, and state-parameter cross-covariance. They note that a single-Gaussian propagation (Algorithm 1) becomes unstable over time.
Resampled Moment Propagation: To address instability and non-Gaussianity, the authors propose a particle-based method (Algorithm 3). This method represents the state distribution as a cloud of particles. Between resampling steps, each particle evolves according to the linearized moment propagation equations, accumulating local covariance. At periodic intervals, the local Gaussian distributions associated with each particle are used to generate a new point cloud via resampling, after which local covariances are reset. This prevents unbounded variance growth while retaining non-Gaussian features. Two resampling strategies are discussed: drawing one sample per Gaussian component and sampling from a Gaussian Mixture Model (GMM).

Key Contributions

Analytical Formulations: Derivation of exact analytical expressions for the PDF and characteristic function of neural network outputs under Leaky ReLU activation, including closed-form approximations for mean and covariance that incorporate parameter uncertainty.
State-Parameter Cross-Covariance: Development of recursive equations that explicitly track the cross-covariance between state and network parameters, a factor often neglected but shown to be essential for accurate multi-step uncertainty propagation.
Stable Particle-Based Algorithm: Introduction of "Resampled Moment Propagation," a hybrid method combining local analytical moment propagation with periodic resampling. This approach maintains stability over long forecast horizons where single-Gaussian methods fail.
High-Dimensional Applicability: Demonstration of the framework on high-dimensional systems (up to 200 dimensions) with complex dynamics.

Results
The framework was tested on two chaotic dynamical systems:

Lorenz-63 System: The Resampled Moment Propagation method closely reproduced the mean and covariance of a Monte Carlo baseline (using 3000 samples) through the predictability horizon ( $T_p \approx 5.11$ ). The method achieved comparable accuracy with as few as 100 particles. The single-Gaussian propagation was found to be unstable, with variance growing rapidly and exceeding the Monte Carlo spread.
Kuramoto–Sivashinsky Equation: Applied to a 200-dimensional discretization of this PDE. The method accurately tracked the mean and initial variance growth over 500 time steps (a fraction of the predictability horizon). The primary limitation identified was memory usage; storing the state-parameter cross-covariance for 100 particles in a 200-dimensional system with ~300,000 parameters required approximately 48 GB of storage. Neglecting this cross-covariance led to substantial underestimation of propagated uncertainty.

Significance and Claims
The paper claims that the proposed framework provides a computationally efficient alternative to full Monte Carlo sampling for uncertainty propagation in auto-regressive neural networks. By leveraging the piecewise-linear nature of Leaky ReLU, the method avoids the need for repeated sampling of the network during the propagation phase, instead relying on analytical Jacobian-based updates. The authors assert that their approach successfully propagates uncertainty through the predictability horizon for chaotic systems and remains applicable to high-dimensional dynamics.

The authors emphasize that the resampling interval is the primary parameter controlling performance: too short an interval leads to particle degeneracy and underestimation of uncertainty, while too long an interval allows local covariances to grow beyond the linearization regime, causing instability. The work highlights the necessity of tracking state-parameter cross-covariance for accurate long-term forecasting and suggests that future improvements could focus on reduced-order representations to mitigate memory costs in large-scale systems. The framework is presented as a pure-forecasting tool, distinct from Bayesian filtering approaches that include measurement updates.

Technical Summary: Uncertainty Propagation in Auto-Regressive Random Neural Network Models

More like this