📊 statistics

A numerical study into neural network surrogate model performance for uncertainty propagation

This study evaluates neural network surrogate models for uncertainty propagation in stochastic heat conduction, revealing that while they accurately predict mean fields, they suffer from significantly larger errors at distribution tails due to extrapolation, with fully connected networks trained on weak form residuals demonstrating superior performance in handling these extreme cases.

Original authors: Noah Wade, Kirubel Teferra

Published 2026-05-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Noah Wade, Kirubel Teferra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to predict the weather. You give it a map of a city and ask it to guess the temperature at every street corner. If you only show the robot sunny days, it gets pretty good at predicting sunny weather. But what happens when you ask it to predict a rare, extreme heatwave or a sudden, freezing blizzard?

This is exactly the problem Noah Wade and Kirubel Teferra tackled in their study. They wanted to see how well "neural network surrogates" (which are like super-smart, fast-acting robot assistants that mimic complex physics) handle uncertainty, specifically when things get extreme.

Here is a breakdown of their findings using simple analogies:

The Setup: The "Thermal Bakery"

The researchers set up a digital simulation of a giant, flat baking sheet (a heat conduction problem). They wanted to see how heat spreads across this sheet when the heat source (the oven element) is unpredictable and changes wildly.

The Challenge: They created two types of "ovens." One had mild, predictable heat variations. The other had wild, chaotic heat spikes that could get extremely hot in very specific spots.
The Goal: Train three different types of "robot chefs" to predict the temperature of the baking sheet for any random oven setting, including the crazy, extreme ones.

The Three Robot Chefs

They tested three different types of neural networks (the "chefs"):

The Data-Driven Chef (DDN): This chef only learns by looking at a massive photo album of past baking results. It memorizes patterns but doesn't know the laws of physics.
The Physics-Informed Chef (PINN): This chef looks at the photo album and carries a textbook on the laws of heat transfer. It is forced to make predictions that obey the laws of physics, even if it hasn't seen that exact situation before.
The Deep Operator Chef (DeepONet): This is a more complex chef designed to understand the relationship between the shape of the heat source and the resulting temperature, rather than just memorizing points.

The Big Discovery: The "Tail" Problem

The researchers found that all three chefs were great at predicting "average" days. If the oven was a normal temperature, the robots were accurate.

However, when they tested the robots on extreme outliers (the "tails" of the distribution—the rare, super-hot heatwaves), the robots struggled.

The Analogy: Imagine a student who studies hard for a math test. They get an A on questions about adding and subtracting. But when the teacher asks a question about a concept the student has never seen before (extrapolation), the student guesses wildly.
The Result: The errors on these extreme samples were ten times larger than the errors on normal samples. The robots tried to guess what would happen outside the range of what they had been trained on, and they often got it wrong.

The "Outlier" Trap

The study discovered a specific reason for these big errors. The robots failed most often when they had to predict a temperature higher than the hottest temperature they had ever seen during training.

The "Convex Hull" Metaphor: Imagine the training data is a group of people standing in a circle. If you ask the robot to predict something for a person standing inside the circle, it's easy. If you ask it to predict for someone standing outside the circle, the robot is just guessing.
The Finding: The robots didn't necessarily fail because the input (the oven setting) was weird; they failed because the result (the temperature) was higher than anything they had ever experienced.

Did They Fix It? (The Weighted Training)

The researchers tried a clever trick called "Weighted Training."

The Idea: They told the robot, "Ignore the easy, average days. Focus 100% of your energy on the rare, extreme heatwaves." They gave the extreme examples more "points" in the grading system.
The Outcome: It worked on the training data! The robot got much better at the extreme examples it was studying.
The Catch: When they tested the robot on new extreme examples it hadn't seen before, it actually got worse. By focusing too hard on the specific extreme examples in the training set, the robot "overfit" (memorized the specific cases) and lost its ability to generalize to new, unseen extremes.

The Winner

Among the three chefs, the Physics-Informed Chef (PINN) performed the best.

Even though it still struggled with the most extreme outliers, it was more reliable than the others.
The "Data-Driven Chef" (who only memorized photos) and the "Deep Operator Chef" (the complex one) both made significantly larger mistakes on the extreme samples.

The Bottom Line

This study is a warning label for using AI in engineering and science.

AI is great at the average: If you just need to know what happens in a typical scenario, neural networks are fast and accurate.
AI is risky at the extremes: If you need to know what happens during a rare disaster or an extreme event, standard AI models might fail spectacularly because they haven't "seen" those scenarios before.
The Solution isn't just "more data": Simply giving the AI more examples of extreme events doesn't always fix the problem if those events are truly outside the range of what the AI has learned. The researchers suggest that instead of just throwing more random data at the problem, we need smarter ways to identify and specifically train on these "outlier" scenarios.

In short: Neural networks are excellent students for the material they've studied, but they are terrible at guessing the answers to questions that are completely outside the textbook.

Technical Summary: A Numerical Study into Neural Network Surrogate Model Performance for Uncertainty Propagation

Problem Statement
Neural network (NN) surrogate models have emerged as a promising tool for solving stochastic boundary value problems (SBVPs), offering the potential to circumvent the repeated, expensive evaluations of traditional numerical solvers required for uncertainty propagation. However, existing literature predominantly focuses on the ability of these models to represent deterministic samples or mean field solutions, often overlooking performance in the tails of the probability distribution. This is a critical gap, as reliability analysis and risk assessment depend heavily on accurate predictions of rare, extreme events. The specific challenge addressed in this study is the "long tail" problem: neural networks typically excel at interpolation but struggle with extrapolation. In SBVPs, the scarcity of training samples in low-probability regions (the tails) leads to diminished accuracy for extreme solution fields, where errors can be an order of magnitude larger than mean field errors.

Methodology
The study evaluates the performance of three distinct neural network architectures in predicting the solution field of a steady-state heat conduction equation with a highly stochastic source term. The governing physics are incorporated via a weak-form residual loss. The problem is defined on a two-dimensional domain where the source term is generated via a Karhunen–Loève decomposition of a random field, followed by a nonlinear transformation to induce heavy-tailed temperature distributions. Two datasets (Source 1 and Source 2) are generated, with Source 2 exhibiting a significantly larger temperature range and more concentrated peaks.

The three models evaluated are:

Data-Driven Network (DDN): A fully connected feed-forward neural network trained exclusively on labeled finite element data.
Physics-Informed Neural Network (PINN): A fully connected network trained using a loss function derived from the weak-form residual of the governing partial differential equation (PDE), without labeled data.
Deep Operator Network (PCA-DeepONet): A physics-informed model utilizing a DeepONet architecture, which separates the learning of the input function (branch network) from spatial coordinates (trunk network).

The study employs a standardized training protocol using identical hyperparameters, collocation points, and training/test splits to ensure a fair comparison. Performance is evaluated using standard metrics (relative $\ell_2$ -norm) and, crucially, higher-order norms including the maximum per-sample error (MPSE) and maximum error (ME) to capture tail behavior. The study also investigates instance-based adaptive weighting strategies to bias training toward outlier samples and analyzes the correlation between prediction errors and specific types of outliers (extrapolation of input parameters vs. extrapolation of solution quantities).

Key Results

Tail Performance Disparity: The worst-case prediction errors for all models were found to be approximately ten times larger than the mean field error. This confirms that standard training metrics (like mean $\ell_2$ error) can mask significant failures in predicting extreme events.
Model Comparison: Among the architectures tested, the PINN (fully connected with physics-informed loss) consistently outperformed both the DDN and the PCA-DeepONet on the test data, particularly regarding the reduction of ME and the distribution of MPSE values. While the PCA-DeepONet performed well on training data, it exhibited significantly higher errors on test data, suggesting poor generalization for this specific stochastic mapping.
Extrapolation Limits: The largest errors were strongly correlated with samples requiring extrapolation beyond the bounds of the training data. Specifically, errors were highest when the predicted maximum temperature of a test sample exceeded the maximum temperature observed in the training set. Interestingly, samples with input parameters lying outside the convex hull of the training set did not necessarily produce large errors; rather, the extrapolation of the response quantity of interest (the solution field) was the primary driver of error.
Weighting Strategies: An adaptive weighting strategy, designed to assign higher weights to samples with larger losses (tail samples), successfully reduced training error. However, this approach led to overfitting, resulting in an increase in testing error. This suggests that simply reweighting existing data is insufficient to solve the extrapolation problem and may degrade generalization.
Data Size Sensitivity: Increasing the number of training samples improved generalization error up to a point, but performance plateaued when the training set no longer contained samples with maximum temperatures exceeding those in the test set.

Significance and Claims
The authors claim that the primary contribution of this work is a rigorous numerical investigation evaluating ubiquitous neural network architectures and loss functions specifically for uncertainty propagation under highly variable solution fields. The study highlights that:

Outlier Identification is Critical: The most significant source of error is the model's inability to extrapolate to extreme solution fields. The paper proposes a method for identifying these "outlier" samples by flagging test cases where the predicted response quantities exceed those seen during training.
Limitations of Current Methods: Standard physics-informed and data-driven approaches, while effective for mean fields, struggle with the tails of the distribution. The study demonstrates that adaptive weighting alone is not a panacea and may lead to overfitting.
Future Direction: The authors suggest that future efforts should focus on using outlier identification to guide adaptive data augmentation—specifically, simulating full-order models for identified outlier regions to construct targeted training datasets—rather than relying on naive random sampling or simple reweighting.

The paper concludes that while neural network surrogates show promise, their application to reliability problems requires careful attention to extrapolation capabilities and the establishment of error bounds for low-probability regions.