Technical Summary: Meta-learning Structure-Preserving Dynamics

Problem Statement

Scientific machine learning (SciML) aims to model complex physical systems, ranging from fluid dynamics to materials science. While physics-informed neural networks (PINNs) incorporate physical biases, they often rely on soft constraints that fail to strictly preserve structural properties (e.g., conservation laws) outside the training distribution, leading to poor generalization and violations of fundamental physical principles.

Structure-preserving neural networks, such as Hamiltonian Neural Networks (HNNs) and GENERIC/Metriplectic Neural Networks, embed physical inductive biases directly into their architectures to guarantee the conservation of energy or the correct generation of entropy. However, these models typically require complete knowledge of system parameters (e.g., mass, damping, stiffness) and are trained on a per-configuration basis. When system parameters vary, these models necessitate costly retraining for each new configuration, creating inefficiencies in "many-query" applications.

While meta-learning offers a potential remedy by learning to adapt to new tasks, existing optimization-based approaches (e.g., MAML, Reptile) suffer from limited generalizability, high computational costs due to gradient-based updates on high-dimensional parameters, and potential instability. Furthermore, there has been no thorough comparison of these optimization-based methods against modulation-based meta-learning techniques in the context of structure-preserving dynamics.

Methodology

The paper proposes a modulation-based meta-learning framework for learning conservative and dissipative dynamical systems with unknown parametric dependence. Instead of optimizing high-dimensional model weights for each new system configuration, the approach conditions a learnable dynamics model on low-dimensional latent variables.

Core Architecture

The framework utilizes a shared set of base parameters ($\Theta_{base}$) across all system instances and a set of individual parameters ($\Theta_{indv}^{(k)}$) specific to each system $k$. The individual parameters are inferred from a latent code $z^{(k)}$ via a hypernetwork $f_{hyper}(z^{(k)}; \phi)$. This allows for "pre-training" of bulk characteristics shared across systems and computationally inexpensive "fine-tuning" for specific instances.

Novel Modulation Techniques

The authors introduce two novel modulation strategies applied to the weights and biases of the underlying Multi-Layer Perceptron (MLP) representing the Hamiltonian or GENERIC energy functions:

1. Latent Multi-Rank (MR) Modulation: Inspired by Low-Rank Adaptation (LoRA), this technique applies layer-wise corrections using low-rank matrices $U^{(\ell,k)}$ and $V^{(\ell,k)}$ generated by the hypernetwork.
   $$h \mapsto \sigma((W^{(\ell)} + U^{(\ell,k)}V^{(\ell,k)T})h + b^{(\ell)} + s^{(\ell,k)})$$
2. Latent SVD-like Modulation: A more parameter-efficient approach resembling Singular Value Decomposition (SVD). It updates weights using a sum of rank-1 updates defined by learned singular values $d_i$ and fixed base basis vectors $u_i, v_i$ (which are part of $\Theta_{base}$).
   $$h \mapsto \sigma((W^{(\ell)} + \sum_{i=1}^r d_i^{(\ell,k)} u_i^{(\ell)} v_i^{(\ell)T})h + b^{(\ell)} + s^{(\ell,k)})$$
   This method imposes soft orthogonality constraints on the basis vectors to ensure distinct modulation directions.

Training and Testing Algorithm

The training process follows a two-loop meta-learning strategy:

• Inner Loop (Auto-decoding): Updates the latent codes $z^{(k)}$ for specific system instances using a small set of trajectories to minimize the velocity matching loss (symplecticity loss for Hamiltonian systems or GENERIC loss for dissipative systems).
• Outer Loop: Updates the base parameters $\Theta_{base}$ and hypernetwork parameters $\phi$ using meta-gradients.
• Testing: Involves a few-shot update of the latent code $z$ for a new target system using a small number of trajectories, while keeping the base parameters fixed. The authors propose initializing test latent codes with the Euclidean mean of training latent codes ($z_{avg}$) rather than zero vectors to improve stability.

Structure Preservation

The paper proves (Proposition D.1) that modulating the parameters of Hamiltonian and GENERIC networks preserves the structural guarantees (energy conservation and thermodynamic laws) regardless of the modulation technique, provided the modulation is applied to the functional parameters defining the energy/entropy functions.

Key Contributions

1. Systematic Investigation: The paper systematically investigates modulation-based techniques for the meta-learning of provably structure-preserving dynamics with unknown parametric dependence.
2. Novel Techniques: Development of two new modulation strategies (Latent MR and Latent SVD-like) that offer improved expressiveness and parameter efficiency over existing methods like Latent Shift and Full-Weight modulation.
3. Extensive Benchmarking: Comprehensive experiments on benchmark problems, including three energy-conserving systems (Mass-Spring, Pendulum, Duffing Oscillator) and one dissipative system (Damped Nonlinear Oscillator).

Results

The proposed methods were compared against standard training ("Scratch"), optimization-based meta-learning (MAML, Reptile, ANIL), and existing modulation baselines (Latent Shift, Latent Full-Weight/FW).

• Performance: Modulation-based algorithms consistently outperformed optimization-based methods, achieving approximately 65% lower errors in field and trajectory metrics across all benchmark problems.
• Best Performers: Among the modulation techniques, SVD-like decompositions (particularly SVD with rank 5, denoted SVD(5)) achieved the lowest errors and highest structural similarity (SSIM) scores.
• Parameter Efficiency: SVD-like modulation achieved superior performance with significantly fewer additional parameters compared to Full-Weight modulation, demonstrating a favorable balance between expressivity and parameter cost.
• Few-Shot Adaptation: The methods demonstrated robust few-shot adaptation, achieving low errors even with a small number of shots (e.g., 5–10 trajectories) for latent code updates.
• Dissipative Systems: In the dissipative case (DNO), optimization-based methods (Reptile, ANIL) often failed to train (producing NaN or diverging results), whereas modulation-based approaches (specifically RO and SVD(3)) remained stable and effective.
• Multi-Domain Generalization: When trained on a combined dataset of multiple dynamical systems, SVD-like modulations successfully adapted a single base network to fundamentally different dynamics, outperforming other baselines.

Significance and Claims

The paper claims to provide a scalable foundation for data-driven modeling of real-world physical systems where parameters vary but structural laws (conservation of energy, entropy production) must be strictly preserved. By sidestepping the issues of explicit optimization for new parameters, the proposed modulation-based meta-learning framework enables accurate few-shot adaptation without compromising the conservation of key invariants.

The authors emphasize that their approach does not require explicit knowledge of the system's parametric dependence during the meta-learning phase, making it suitable for scenarios where physical parameters are unknown or difficult to measure. The work contributes to the field of SciML by advancing meta-learning methods specifically tailored for structure-preserving dynamical systems, addressing the limitations of both standard training and existing optimization-based meta-learning strategies.