Variational meta-learning inference for low dimensional neural system identification

Technical Summary: Variational Meta-Learning Inference for Low-Dimensional Neural System Identification

Problem Statement
Deep learning has become a dominant tool for nonlinear system identification due to its ability to model complex dynamics without explicit physical laws. However, heavily parameterized neural networks suffer from overfitting and poor generalization in low-data regimes, lacking reliable uncertainty quantification. While physics-based models offer data efficiency, deriving them for complex systems is often infeasible. Recent manifold meta-learning frameworks (specifically [9]) address data efficiency by restricting neural network parameters to a meta-learned, low-dimensional manifold. However, the existing framework is purely deterministic, mapping datasets to single point estimates of parameters, thereby failing to quantify the epistemic uncertainty inherent in scarce data scenarios. This limitation is critical for safety-critical applications where understanding the confidence in parameter estimates and predictions is paramount.

Methodology
The paper proposes a fully probabilistic extension of the manifold meta-learning framework based on amortized Variational Inference (VI). The core methodology involves the following components:

1. Generative Probabilistic Model: The authors introduce a generative prior $p_\omega(\phi)$ over the low-dimensional latent parameters $\phi$ and a likelihood function $p_\gamma(y|u, \phi)$. The likelihood is modeled as a Gaussian distribution where the mean is determined by the lifted neural network output $F(u, P_\gamma(\phi))$, and the variance represents observation noise.
2. Amortized Variational Inference: To approximate the intractable posterior $p(\phi|D)$, an inference network (encoder) $E_\psi$ is trained to map a training dataset $D_{tr}$ to the sufficient statistics (mean and diagonal covariance) of an approximate Gaussian posterior $q_\psi(\phi|D_{tr})$.
3. Training Objective (Meta-ELBO): The meta-parameters (generative model parameters $\gamma$ and encoder weights $\psi$) are optimized by maximizing the Evidence Lower Bound (ELBO). The ELBO balances the expected log-likelihood (reconstruction error on the test split) and a Kullback-Leibler (KL) divergence term that regularizes the approximate posterior against the prior (an isotropic Gaussian).
4. Meta-Testing and Uncertainty Quantification:
   • Initialization: At test time, the encoder provides a zero-shot Gaussian approximation of the posterior.
   • MAP Refinement: To improve accuracy, the encoder's mean output serves as an initialization for a Maximum A Posteriori (MAP) estimation, refining the latent parameter $\phi$ via gradient descent on the negative log-posterior.
   • Laplace Approximation: To obtain a rigorous uncertainty measure, the authors apply a Laplace approximation around the MAP estimate. The posterior covariance is approximated as the inverse Hessian of the negative log-posterior loss. This is computationally feasible because the dimensionality of the latent manifold ($n_\phi$) is small.

Key Contributions

• Probabilistic Extension: The primary contribution is the transformation of the deterministic manifold meta-learning framework [9] into a probabilistic setting using Variational Autoencoder (VAE) principles.
• Calibrated Uncertainty: The method provides mathematically grounded uncertainty bounds for both the low-dimensional latent parameters and the resulting system predictions.
• Preservation of Accuracy: The approach achieves predictive accuracy comparable to the deterministic counterpart while adding uncertainty quantification capabilities.
• Laplace Approximation for Manifold Learning: The paper demonstrates the feasibility of using the Laplace approximation on the low-dimensional manifold to yield fast, principled uncertainty estimates without sacrificing the efficiency of amortized inference.

Results
The framework was evaluated on two benchmarks: a static regression task involving sinusoidal functions and the Bouc–Wen dynamical system benchmark (a hysteretic mechanical system).

• Static Regression (Sines): In a low-data regime (as few as 1 training point), the probabilistic model maintained the learned sinusoidal structure and provided tight uncertainty bounds that correctly reflected the scarcity of data. The predictive Root Mean Square Error (RMSE) remained competitive with the deterministic baseline.
• Bouc–Wen Benchmark:
  • Predictive Performance: The variational framework achieved test FIT indices comparable to the deterministic manifold meta-learning approach across various training sequence lengths (from 100 to 5000 samples).
  • Uncertainty Calibration: The Laplace approximation yielded well-calibrated confidence intervals. Empirical coverage analysis showed that the $3\sigma$ bounds closely matched the theoretical 99.7% coverage.
  • Data Scaling: The predictive standard deviation exhibited a monotonic decay as the amount of adaptation data increased, aligning with Bayesian principles where epistemic uncertainty shrinks with more information.
  • Prior Validation: Unconditional sampling from the learned prior generated plausible system trajectories, confirming the model successfully captured the underlying generative process.

Significance and Claims
The paper claims that this work successfully bridges the gap between data-efficient manifold meta-learning and rigorous uncertainty quantification. By extending the deterministic framework to a probabilistic setting, the authors demonstrate that it is possible to retain the high predictive performance of the original method while gaining the ability to quantify epistemic uncertainty. This is particularly significant for system identification in low-data regimes where traditional deep learning models fail to generalize or provide confidence metrics.

The authors explicitly note limitations: the Laplace approximation is a local approximation and may not capture multimodal posteriors (e.g., in systems with multiple parameter combinations producing similar outputs, such as the Bouc–Wen system under specific excitations). However, they argue that for the tested regimes, the approach provides a robust, mathematically grounded solution. The work positions itself as a step toward integrating data-driven uncertainty bounds into risk-aware control applications, such as stochastic Model Predictive Control, though the paper itself does not implement these control applications.