SPIKE: Sparse Koopman Regularization for Physics-Informed Neural Networks

1. Problem Statement

Physics-Informed Neural Networks (PINNs) have emerged as a mesh-free method for solving partial differential equations (PDEs) by embedding physical laws (residuals) directly into the loss function. However, standard PINNs face two critical limitations:

1. Poor Out-of-Distribution (OOD) Generalization: PINNs tend to overfit within the training spatiotemporal domain. When extrapolating to unseen time horizons or spatial regions, errors grow rapidly, leading to catastrophic failure, particularly in stiff, chaotic, or high-order systems.
2. Lack of Interpretability: The learned weights of the neural network are opaque, failing to reveal the underlying parsimonious (simple) dynamical structure of the physical system.

Existing approaches to improve generalization (e.g., variance-based regularization) or identify dynamics (e.g., SINDy, Deep Koopman) often require explicit knowledge of the system dynamics $f(x)$ or fail to address the structural instability of PINNs during extrapolation.

2. Methodology: The SPIKE Framework

The authors propose SPIKE (Sparse Physics-Informed Koopman-Enhanced), a framework that augments standard PINNs with continuous-time Koopman operator regularization. Unlike prior works that use Koopman theory as the primary solver, SPIKE uses it as an auxiliary regularizer to enforce structural constraints on the learned dynamics.

Core Components:

• Augmented Embedding: The solution field $u(x,t)$ is lifted into an observable space $z = g(u)$ via a dual-component embedding:
  • Library Component ($g_{lib}$): A learnable projection of explicit polynomial terms (e.g., $1, u, u^2, \dots$).
  • Latent Component ($g_{mlp}$): An MLP that learns implicit features, capturing correlations with derivatives (e.g., $u_{xx}$) even without explicit derivative terms in the library.
• Continuous-Time Koopman Dynamics: The framework enforces linear dynamics in the observable space:
  $$\frac{dz}{dt} = Az$$
  where $A$ is the infinitesimal generator matrix. This contrasts with discrete-time formulations ($z_{t+\Delta t} = K z_t$) which suffer from "identity collapse" (off-diagonal elements vanish as $\Delta t \to 0$) and diagonal dominance issues.
• Sparsity Regularization (SPIKE): To enforce the parsimony principle (that complex dynamics often admit low-dimensional structures), an $L_1$ penalty is applied to the generator matrix $A$:
  $$\mathcal{L}_{sparse} = \lambda_s \|A\|_1$$
  This forces $A$ to be sparse, identifying the minimal set of active terms driving the system.
• Matrix Exponential Integration: For stiff systems (e.g., Cahn-Hilliard, Kuramoto-Sivashinsky), the framework utilizes matrix exponential integration ($z(t+\Delta t) = e^{A\Delta t}z(t)$). This provides unconditional stability, avoiding the training collapse common in explicit Euler methods for stiff PDEs.

Training Objective:

The total loss combines the standard PDE residual, boundary/initial conditions, Koopman consistency, and sparsity:
$$\mathcal{L}_{total} = \mathcal{L}_{physics} + \lambda_k \mathcal{L}_{koopman} + \lambda_s \|A\|_1 + \mathcal{L}_{IC/BC}$$
The Koopman loss $\mathcal{L}_{koopman}$ is computed by comparing the propagated state (via Euler, RK4, or Matrix Exponential) against the actual network output at the next timestep.

3. Key Contributions

1. Koopman Regularization for Generalization: The sparse Koopman constraint acts as a dynamics-aware regularizer. It prevents PINNs from learning spurious high-frequency modes that fit training data but fail to extrapolate.
2. Parsimonious Dynamics Representation: By enforcing $L_1$ sparsity on $A$, the method reduces non-zero generator entries by up to 5.7$\times$, yielding interpretable models that align with the minimal set theorem of Koopman theory.
3. Library-Latent Decomposition: The architecture successfully learns implicit derivative structures (e.g., $u_{xx}$) within the latent MLP, achieving correlations up to 0.99 with true derivatives, enabling coefficient recovery without explicit derivative libraries.
4. Continuous-Time Formulation: The direct learning of the generator $A$ avoids the timestep-dependent instability of discrete Koopman operators, making the method robust for stiff systems.
5. Post-Hoc Inverse Problem Solving: The framework enables the recovery of unknown PDE coefficients with $<1\%$ error after training on forward problems, leveraging the smooth gradients of the PINN.

4. Experimental Results

The framework was evaluated on 14 dynamical systems, including 1D/2D PDEs (Heat, Advection, Burgers, Navier-Stokes, Cahn-Hilliard, Kuramoto-Sivashinsky) and chaotic ODEs (Lorenz).

• Temporal Extrapolation: SPIKE/PIKE demonstrated massive improvements in valid prediction time.
  • Lorenz System: Achieved a 184$\times$ improvement in valid prediction time (12.91s vs. 0.07s for PINN), extending predictions from <0.1 Lyapunov times to ~12 Lyapunov times.
  • Stiff PDEs: For the Cahn-Hilliard equation, PIKE achieved a $10^6\times$ reduction in in-domain MSE compared to PINN.
• Spatial Generalization: For open-domain systems (e.g., 2D Navier-Stokes channel flow), SPIKE reduced OOD spatial MSE by 32$\times$, enabling physically meaningful downstream predictions.
• Stability: The Matrix Exponential (EXPM) integrator provided unconditional stability for stiff systems where standard PINNs and explicit integrators (Euler) failed completely.
• Sparsity & Interpretability:
  • $L_1$ regularization reduced non-zero entries in $A$ significantly (e.g., Schrödinger: 17 $\to$ 3 non-zeros).
  • The learned sparse matrices correctly identified active polynomial terms (e.g., $u \cdot u_x$ in Burgers, $u_{xx}$ in Heat).
• Coefficient Recovery: Post-hoc regression on PINN derivatives recovered PDE coefficients (e.g., diffusion coefficients, reaction rates) with <1% relative error across all tested systems.

5. Significance and Impact

• Solving Fundamental PINN Limitations: SPIKE addresses the root cause of PINN failure in extrapolation: the lack of structural constraints on the solution manifold. By enforcing linear dynamics in a lifted space, it ensures the learned solution respects the global geometry of the attractor.
• Bridging Data-Driven and Physics-Driven: The method unifies the mesh-free flexibility of PINNs with the interpretability and stability of Koopman operator theory. It allows for "black-box" neural networks to produce "white-box" sparse dynamical models.
• Robustness to Stiffness: The continuous-time formulation with matrix exponential integration solves a long-standing issue in scientific machine learning: training neural networks on stiff, high-order PDEs without numerical instability.
• Practical Utility: The ability to perform inverse problems (coefficient recovery) without retraining and to generalize to unseen domains makes this framework highly applicable to real-world scientific computing, fluid dynamics, and chaotic system prediction.

In conclusion, SPIKE represents a paradigm shift from simply minimizing PDE residuals to regularizing the learned dynamics with sparse, continuous-time Koopman operators, resulting in neural networks that are not only accurate but also stable, interpretable, and capable of reliable extrapolation.