Enhancing neural network extrapolation in thermo-fluid systems using steady-state solutions

Technical Summary: Enhancing Neural Network Extrapolation in Thermo-Fluid Systems Using Steady-State Solutions

Problem Statement
Time-dependent partial differential equations (PDEs) governing thermo-fluid systems, such as those found in power transformers and gas turbines, often require computationally expensive long-time simulations. Classical numerical methods (finite difference, volume, or element) rely on sequential time-stepping constrained by stability and accuracy, making the resolution of long-time dynamics prohibitive when the system's relaxation timescale far exceeds the allowable time step. While Physics-Informed Neural Networks (PINNs) offer a mesh-free alternative, standard architectures frequently fail to generalize beyond their training time interval. They lack the necessary inductive bias to represent asymptotic decay, leading to rapid degradation in accuracy during temporal extrapolation. This limitation is critical for engineering applications where capturing the full evolution from initial conditions to a stationary equilibrium is necessary but costly.

Methodology
The authors propose a "steady-state-informed" neural network representation designed for dissipative PDE systems that relax toward a stationary equilibrium. The core innovation is a solution ansatz that decomposes the predicted solution $u(x, t)$ into two components:
$$u(x, t) = u_s(x) + f(t)g(x, t)$$
where:

• $u_s(x)$ is the steady-state component, representing the asymptotic equilibrium.
• $f(t)$ is a time-dependent decay profile (e.g., exponential, algebraic, or damped oscillatory) satisfying $\lim_{t\to\infty} f(t) = 0$.
• $g(x, t)$ is a transient correction term, bounded in space and time.

This decomposition embeds the convergence to the steady state directly into the network architecture rather than enforcing it via a penalty term in the loss function. The steady-state component $u_s(x)$ can be supplied from classical solvers, experimental data, or a dedicated sub-network. The decay profile $f(t)$ can be fixed or learned as a trainable parameter (e.g., decay rate $\lambda$).

The implementation utilizes a Physics-Informed Neural Network (PINN) framework trained with the SOAP (Shampoo with Adam in the Preconditioner's Eigenbasis) optimizer. SOAP is selected to address the ill-conditioned, multi-objective nature of PINN loss landscapes by improving gradient conditioning and reducing directional conflicts between loss terms. The training follows a two-phase strategy:

1. Phase 1 (Steady-State): Train sub-networks to approximate $u_s(x)$ using data or steady-state physics residuals.
2. Phase 2 (Transient): Freeze the steady-state weights and train the remaining sub-networks to learn the transient correction $g(x, t)$ using time-dependent physics-informed losses.

Key Contributions

• Architectural Inductive Bias: The paper introduces a modular architecture that enforces asymptotic convergence structurally, allowing the network to learn transient dynamics while guaranteeing correct long-time behavior.
• Optimization Strategy: The application of the SOAP optimizer to PINNs for thermo-fluid problems, demonstrating improved convergence over standard first-order methods like Adam.
• Temporal Profile Analysis: A systematic evaluation of different temporal decay profiles (constant, exponential, algebraic, damped oscillatory) to determine their impact on extrapolation accuracy across varying physical regimes.

Results
The method was evaluated on a sequence of problems with increasing complexity:

1. 1D Heat Equation: The proposed method significantly outperformed standard PINNs in extrapolation. An exponential decay profile ($f(t) = e^{-\lambda t}$) yielded the lowest error, aligning with the eigenmode structure of the heat equation. Standard PINNs and the constant profile baseline showed monotonically increasing errors during extrapolation.
2. 2D Lid-Driven Cavity (Navier-Stokes): The approach successfully extrapolated velocity fields beyond the training horizon. Exponential and damped oscillatory profiles performed best, with errors remaining bounded and physically consistent even at times three times the training duration.
3. 2D Natural Convection (Boussinesq): In an advection-dominated regime, the damped oscillatory profile ($f(t) = e^{-\lambda t}(a + \cos(\omega t))$) outperformed the pure exponential. The additional amplitude parameter $a$ allowed the model to decouple the initial transient magnitude from the decay rate, accommodating the rapid initial fluctuations before settling.
4. 3D Conjugate Heat Transfer: The method was applied to a complex 3D system with fluid-solid interfaces. Two training scenarios were tested:
   • Pure PINN: Training all components with physics-informed losses resulted in convergence to an erroneous steady state due to inaccuracies in the learned steady-state component.
   • Hybrid Approach: Training the steady-state sub-networks ($FNN_1, FNN_3$) using high-fidelity Finite Element Method (FEM) data, while training transient components with PINN losses, drastically reduced extrapolation error. This confirmed that the accuracy of the final solution is bounded by the accuracy of the supplied steady-state component.

Significance and Claims
The paper claims that the primary contribution is not a new loss function or penalty term, but a steady-state-informed representation that integrates asymptotic behavior directly into the neural network architecture. The authors assert that this approach substantially improves temporal extrapolation compared to architectures that do not explicitly enforce the asymptotic condition.

The work highlights that while standard PINNs struggle with long-time dynamics, exploiting the known structural property of dissipative systems (convergence to equilibrium) allows for the creation of robust surrogates. The authors note that the method is particularly effective when the steady-state solution is known or can be computed cheaply, serving as an anchor for predicting transient evolution. The study concludes that selecting an appropriate temporal profile is critical and problem-dependent, with exponential and damped oscillatory profiles generally providing the most robust performance for thermo-fluid applications.