mLaSDI: Multi-stage latent space dynamics identification

1. Problem Statement

The paper addresses the challenge of creating accurate Reduced-Order Models (ROMs) for time-dependent Partial Differential Equations (PDEs) using data-driven methods. Specifically, it targets the limitations of Latent Space Dynamics Identification (LaSDI), a framework that combines autoencoders with Sparse Identification of Nonlinear Dynamics (SINDy).

• The Core Conflict: In standard LaSDI, an autoencoder must simultaneously satisfy two competing objectives:
  1. Reconstruction: Accurately reconstructing high-fidelity training data.
  2. Dynamics: Producing latent trajectories that satisfy a user-specified, interpretable Ordinary Differential Equation (ODE) structure (e.g., linear dynamics).
• The Limitation: These competing objectives often force the autoencoder to compromise. Neural networks suffer from spectral bias (a tendency to learn low-frequency functions first), and imposing smooth latent dynamics exacerbates this. Consequently, standard LaSDI struggles to capture high-frequency dynamics or complex multiscale phenomena, leading to significant reconstruction and prediction errors. Achieving acceptable accuracy often requires massive models and extensive hyperparameter tuning.

2. Methodology: mLaSDI

The authors propose Multi-stage Latent Space Dynamics Identification (mLaSDI) to decouple the reconstruction of high-frequency details from the learning of interpretable low-frequency dynamics.

Core Framework

mLaSDI trains a sequence of models (stages) rather than a single autoencoder:

1. Stage 1 (Standard LaSDI):
   • An autoencoder (Encoder $G_{enc,1}$ and Decoder $G_{dec,1}$) is trained to compress data into a latent space $z$.
   • SINDy is applied to learn interpretable ODEs governing the latent dynamics ($\dot{z} \approx \Theta(z)\Xi$).
   • This stage captures the dominant, low-frequency, smooth dynamics but often misses fine-scale features.
2. Subsequent Stages (Residual Learning):
   • The latent trajectories ($\hat{z}$) learned in Stage 1 are fixed. This preserves the interpretability of the dynamics.
   • A new decoder ($G_{dec,k}$) is trained to map these same latent trajectories to the residuals (errors) of the previous stage.
   • Normalization: The residual is normalized by its standard deviation to ensure stable gradient flow.
   • Architecture: Later-stage decoders utilize periodic activation functions (specifically Sine) in the first layer. This is critical for overcoming spectral bias and enabling the network to represent high-frequency content that the first stage missed.
3. Final Reconstruction:
   The final output is the sum of the Stage 1 reconstruction and the scaled corrections from subsequent stages:
   $$\hat{U} \approx \tilde{U}_1 + \sum_{i=2}^{n} \epsilon_{i-1} \tilde{U}_i$$
   Where $\tilde{U}_i$ is the output of the $i$-th decoder and $\epsilon$ are normalization factors.

Theoretical Guarantees

• Error Decomposition: The authors derive a continuous-time error bound showing that total error is the sum of autoencoder reconstruction error and latent dynamics mismatch.
• Monotonicity: They prove (Proposition 2) that under idealized conditions (global minimization at each stage), adding a new stage cannot increase the training residual norm. The training error is guaranteed to decrease or stay the same with each added stage.

3. Key Contributions

1. Decoupling Accuracy and Interpretability: mLaSDI resolves the trade-off in LaSDI by allowing the first stage to focus on interpretable, smooth dynamics while subsequent stages focus purely on reconstructing high-frequency residuals.
2. Spectral Bias Mitigation: By introducing periodic activation functions (Sine) in residual decoders, the framework effectively captures high-frequency content that standard ReLU/Tanh networks miss.
3. Efficiency: The method achieves higher accuracy with fewer training epochs and less hyperparameter sensitivity compared to standard LaSDI. It allows for smaller models to outperform larger, single-stage models.
4. Theoretical Analysis: Provides rigorous error bounds and a proof of residual monotonicity for the multi-stage training process.

4. Numerical Results

The authors validated mLaSDI on three distinct problems, comparing it against standard Gaussian Process-based LaSDI (GPLaSDI):

• Multiscale Oscillating System (Toy Problem):
  • Challenge: A function combining a low-frequency sine wave with a high-frequency cosine wave.
  • Result: Standard LaSDI failed to capture the high-frequency component (errors 4–20%). mLaSDI reduced errors by an order of magnitude (0.1–3%) and captured the high-frequency details perfectly.
• Unsteady Wake Flow (Navier-Stokes):
  • Challenge: Flow past a cylinder with vortex shedding.
  • Result: mLaSDI reduced relative errors by 1–3% compared to LaSDI. Qualitatively, mLaSDI produced more accurate vortex shapes and smoother background fields, whereas LaSDI predictions showed circular artifacts and noise.
• 1D-1V Vlasov Equation (Plasma Instability):
  • Challenge: Two-stream instability with complex phase-space structures.
  • Result: mLaSDI models with fewer parameters outperformed larger LaSDI models. mLaSDI achieved median error reductions of 2.5x to 5.9x compared to LaSDI. Crucially, mLaSDI achieved maximum relative errors below 1%, a threshold LaSDI could not reach even with increased model size.

5. Significance

• Paradigm Shift: mLaSDI demonstrates that one does not need to sacrifice interpretability to achieve high-fidelity reconstruction in data-driven ROMs. By treating the "hard" part of the problem (high-frequency residuals) as a secondary correction task, the primary model remains simple and interpretable.
• Practical Impact: The method reduces the computational cost of training (fewer epochs) and the cost of hyperparameter tuning, making high-accuracy reduced-order modeling more accessible for complex scientific simulations.
• Future Direction: The paper suggests that this multi-stage approach could be extended to incorporate weak-form SINDy for noisy data, non-linear latent dynamics, and physics-informed constraints to improve extrapolation capabilities.

In summary, mLaSDI is a robust framework that enhances the Latent Space Dynamics Identification method by using a staged residual learning approach, effectively overcoming the spectral bias of neural networks and the tension between model interpretability and reconstruction accuracy.