🤖 machine learning

Data-Augmented Predictive Deep Neural Network: Enhancing the extrapolation capabilities of non-intrusive surrogate models

This paper proposes a data-augmented deep learning framework that combines kernel dynamic mode decomposition with a convolutional autoencoder to enhance the extrapolation capabilities of non-intrusive surrogate models, enabling accurate and fast predictions of complex parametric nonlinear dynamical systems beyond their training time intervals and parameter samples.

Original authors: Shuwen Sun, Lihong Feng, Peter Benner

Published 2026-07-27

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Shuwen Sun, Lihong Feng, Peter Benner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to predict the weather for next week, but you only have detailed data for the last three days. In the world of complex physics and engineering, scientists face a similar challenge. They build "high-fidelity models"—super-detailed digital twins of real-world systems like flowing water or electrical signals in the heart. These models are incredibly accurate but also incredibly heavy; running a single simulation can take hours or even days on massive supercomputers. To speed things up, researchers create "surrogate models," which are like lightweight, fast-running shortcuts that guess the answer without doing all the heavy lifting.

However, there's a catch. Most of these shortcuts are great at remembering what they've seen, but they struggle to imagine what comes next. If you train a shortcut using data from time zero to ten seconds, it often gets confused when asked to predict what happens at eleven seconds or later. It's like a student who memorizes a textbook perfectly but fails the exam because the questions are about a chapter the teacher never covered. This paper tackles that specific problem: how to teach a computer model to confidently extrapolate, or "reach out," into the future without needing to run the expensive, slow simulations again.

The Paper's Big Idea: Teaching a Model to Dream of the Future

The authors, Sun, Feng, and Benner, propose a clever new framework they call Data-Augmented Predictive Deep Neural Network (DAPredDNN). Think of this as a three-step magic trick designed to make a fast, cheap computer model predict the future of complex systems accurately, even for times it has never seen before.

Step 1: The Compression (The "Suitcase" Analogy)
First, the system uses a tool called a Convolutional Autoencoder (CAE). Imagine you have a massive, messy room full of furniture (the complex physical data). The CAE is like a genius packer who can fold all that furniture into a tiny, compact suitcase (the "latent space"). This suitcase is so small and simple that a computer can carry it around easily. The CAE learns how to pack the room into the suitcase and, just as importantly, how to unpack it back into a full room later.

Step 2: The Crystal Ball (The "Time Traveler" Analogy)
Here is where the paper gets creative. Usually, if you want to know what happens in the future, you have to simulate it step-by-step, which is slow. Instead, the authors use a mathematical tool called Kernel Dynamic Mode Decomposition (KDMD). Think of KDMD as a crystal ball that looks at the pattern of how the suitcase contents move from time zero to time ten. It figures out the "rhythm" or the "dance steps" of the system. Once it understands the rhythm, it can instantly guess what the suitcase will look like at time twenty, time thirty, or even time one hundred, without needing to walk through every single second in between.

Step 3: The Augmentation (The "Study Guide" Analogy)
This is the secret sauce. The authors take the "future guesses" generated by the crystal ball (KDMD) and mix them back in with the original training data. It's like a student who only studied chapters 1 through 10, but then the teacher hands them a "cheat sheet" that summarizes what chapters 11 through 20 would look like based on the patterns of the first ten. The student then studies this new, "augmented" guide.

Once the model is trained on this expanded guide, it learns a direct connection between any time and any setting (parameter) and the final answer. When you ask the model a question about the future (say, time 15), it doesn't have to guess step-by-step. It just looks at its training and says, "Ah, I know exactly what that looks like!" in a single, lightning-fast step.

What They Found

The team tested this method on two very different problems to see if it actually worked.

The Heartbeat Model (FitzHugh-Nagumo): They simulated a model that mimics how electrical signals travel through heart cells. They trained the model on data up to time $T_0 = 12$ seconds and asked it to predict up to $T = 20$ seconds. The results were impressive: the model predicted the future behavior with tiny errors (less than 2% at the worst points). It successfully captured the complex "limit cycles" (repeating patterns) of the heart cells even though it had never seen that specific time range during its initial training.
The Wind Tunnel (Flow Past a Cylinder): They simulated water or air flowing around a cylinder (like a bridge pillar). They trained the model on data from time 4.5 to 8.1 seconds and asked it to predict the flow up to 9.0 seconds. This included tricky situations where the flow started swirling and shedding vortices (eddies). The model predicted these swirling patterns accurately, even for Reynolds numbers (a measure of flow speed) that were completely new and outside the training range. The average error was incredibly low, around $5.5 \times 10^{-5}$ .

Why This Matters

The paper explicitly argues against methods that rely on "step-by-step" prediction (like taking one step, then using that result to take the next). The authors point out that those methods are slow and tend to accumulate errors the further you go into the future. Their method, by contrast, predicts the entire future sequence in one single step.

They also clarify that this isn't just about being faster; it's about being smarter at extrapolation. By using KDMD to generate "augmented" data, they allow the neural network to learn the future dynamics without ever having to run the expensive, slow simulations for that future time.

In short, the paper suggests that by combining a smart data compressor (CAE), a rhythm-finder (KDMD), and a fast predictor (Deep Neural Network), we can build models that don't just remember the past but can confidently and accurately imagine the future, all in the blink of an eye. The authors conclude that this approach is a reliable way to handle complex, changing systems, opening the door for faster and more efficient simulations in engineering and science.

Technical Summary: Data-Augmented Predictive Deep Neural Network (DAPredDNN)

1. Problem Statement

The paper addresses the challenge of efficiently predicting the dynamics of high-fidelity, parametric nonlinear dynamical systems (governed by time-dependent PDEs) in multi-query scenarios. Direct simulation of these Full Order Models (FOM) is computationally prohibitive. While Reduced Order Models (ROM) and machine learning-based surrogates offer a solution, existing non-intrusive methods often struggle with time extrapolation.

Specifically, many current deep learning approaches (e.g., those using LSTMs or step-by-step predictors) can accurately predict system states within the training time interval $[0, T_0]$ . However, they fail to generalize accurately to the interval $[T_0, T]$ where $T > T_0$ , unless they rely on sequential, iterative predictions (p-step to k-step). This sequential dependency introduces error accumulation and slows down online prediction. Furthermore, methods requiring high-fidelity simulations for the extrapolation interval $[T_0, T]$ during the online phase negate the efficiency gains of surrogate modeling.

2. Methodology: DAPredDNN

The authors propose a Data-Augmented Predictive Deep Neural Network (DAPredDNN). This framework combines a Convolutional Autoencoder (CAE), a Feed-Forward Neural Network (FFNN), and Kernel Dynamic Mode Decomposition (KDMD) to achieve accurate, one-step prediction across both parameter and time domains without requiring FOM simulations for the extrapolation period.

The methodology proceeds in three distinct stages:

A. Architecture Components

Convolutional Autoencoder (CAE): Used for nonlinear dimensionality reduction. The encoder $\Phi$ maps high-dimensional physical states $u_h(\mu, t)$ to a low-dimensional latent space $z(\mu, t) \in \mathbb{R}^n$ ( $n \ll N$ ). The decoder $\Psi$ reconstructs the physical state from the latent variables.
Feed-Forward Neural Network (FFNN): Maps the parameter-time pair $(\mu, t)$ directly to the latent space, learning the relationship $\Gamma(\mu, t) \approx z(\mu, t)$ .
Kernel Dynamic Mode Decomposition (KDMD): A linear operator-based method used to evolve the latent dynamics. Based on Koopman theory, KDMD approximates the evolution function $F$ in the latent space as a linear operator $K$ , allowing for the prediction of future latent states $z(\mu, t^*)$ for $t^* > T_0$ using only data from $[0, T_0]$ .

B. The Three-Step Training Process

The core innovation is the data augmentation strategy used during the offline training phase:

Pretraining CAE: The CAE is trained on the original dataset containing snapshots from training parameters $\{\mu_1, \dots, \mu_k\}$ within the time interval $[0, T_0]$ . This establishes the encoder $\Phi$ and decoder $\Psi$ .
KDMD Extrapolation (Data Generation):
- The pretrained encoder maps the training data to the latent space, creating snapshot matrices $Z(\mu_i)$ for $t \in [0, T_0]$ .
- KDMD is applied to these latent snapshots to predict the evolution of the latent states into the extrapolation interval $[T_0, T]$ , yielding $\tilde{z}(\mu_i, t^*)$ .
- The pretrained decoder reconstructs these extrapolated latent states back into the physical space, generating synthetic data $\tilde{u}_h(\mu_i, t^*)$ for $t^* \in [T_0, T]$ .
Training Augmented CAE-FFNN:
- The original training data $[0, T_0]$ is concatenated with the KDMD-generated extrapolated data $[T_0, T]$ to form a new, augmented dataset covering the full interval $[0, T]$ .
- The CAE and FFNN are jointly trained on this augmented dataset. The loss function combines the CAE reconstruction loss and the FFNN latent prediction loss.
- This process teaches the FFNN to map any $(\mu, t)$ pair directly to the correct latent state, effectively "learning" the extrapolated dynamics without needing the KDMD operator during inference.

C. Online Prediction

During the online phase, the model requires only the trained FFNN and decoder. Given any new parameter $\mu^*$ and any time instant $\hat{t} \in [0, T]$ , the model performs a single-step prediction:
$\tilde{u}_h(\mu^*, \hat{t}) = \hat{\Psi}(\Gamma(\mu^*, \hat{t}))$
This eliminates the need for sequential time-marching or iterative predictions, significantly reducing computational cost.

3. Key Contributions and Novelty

The paper highlights three primary novelties:

KDMD-Enabled Latent Evolution: By employing KDMD in the latent space, the framework captures the system's dynamics beyond the training time horizon $T_0$ without requiring high-fidelity simulations for that period.
Offline Data Augmentation: The method enriches the training dataset with KDMD-decoder-extrapolated data. This allows the neural network to learn the features of the extrapolation phase directly, rather than relying on iterative inference.
One-Step Online Prediction: Unlike existing methods (e.g., LSTM-based or SINDy-based approaches) that require $p$ -step to $k$ -step sequential predictions, DAPredDNN predicts the entire time sequence (or any specific time step) in a single forward pass. This ensures high efficiency and avoids error accumulation associated with sequential prediction.

4. Numerical Results

The method was validated on two numerical examples:

A. FitzHugh–Nagumo Model

Setup: A 1D reaction-diffusion system with parameter $\epsilon \in [0.02, 0.04]$ . Training data covered $[0, 12]$ ( $T_0$ ), while the full interval was $[0, 20]$ ( $T$ ).
Performance: The model successfully predicted solutions for 10 unseen testing parameters across the entire time domain.
Accuracy: The mean relative error ( $\epsilon_{mean}$ ) was approximately $8.15 \times 10^{-4}$ for the membrane potential $v$ and $4.48 \times 10^{-4}$ for the recovery variable $w$ . The maximum errors remained low ( $< 2 \times 10^{-2}$ ).
Observation: The model accurately captured limit cycles and temporal evolution even in the extrapolation region $[12, 20]$ where no training data existed.

B. Incompressible Flow Past a Cylinder

Setup: 2D Navier-Stokes equations with Reynolds number $Re$ as the parameter. Training data covered $Re \in [100, 750]$ and time $[4.5, 8.1]$ . The extrapolation target was time $[8.1, 9.0]$ (25% of the total interval).
Performance: Tested on 6 unseen $Re$ values, including one outside the training range ($Re=800$).
Accuracy: The method accurately predicted steady laminar flows and complex vortex shedding patterns.
- Mean relative error: $5.498 \times 10^{-5}$ .
- Maximum relative error: $1.888 \times 10^{-3}$ .
Observation: The model maintained high accuracy for both interpolation and extrapolation in time, and for parameters both within and outside the training set.

5. Significance and Claims

The paper claims that DAPredDNN provides a robust, non-intrusive solution for parametric model order reduction that overcomes the time-extrapolation limitations of current deep learning surrogates.

Efficiency: The one-step prediction capability makes the online phase extremely fast compared to iterative methods.
Accuracy: The integration of KDMD allows the model to learn the underlying dynamics of the extrapolation phase, resulting in high fidelity even when training data is restricted to a shorter time window.
Applicability: The method is applicable to complex, high-dimensional nonlinear systems (e.g., fluid dynamics) where traditional intrusive MOR is difficult to implement.

The authors note that while the method is effective, future work could involve introducing a discriminator to validate the quality of KDMD-generated data, potentially improving stability and reducing hyperparameter tuning efforts. The code and data are made available for reproducibility.