Invariant Manifolds of Discrete-time Dynamical Systems with Nonlinear Exosystems via Hybrid Physics-Informed Neural Networks

Technical Summary: Invariant Manifolds of Discrete-Time Dynamical Systems with Nonlinear Exosystems via Hybrid Physics-Informed Neural Networks

Problem Statement
The paper addresses the challenge of approximating invariant manifolds (IMs) for discrete-time dynamical systems driven by exogenous autonomous dynamics (exosystems). Such hierarchical or coupled systems are prevalent in control theory (e.g., nonlinear output regulation) and collective behavior modeling (e.g., leader-follower dynamics in traffic, bird flocks, and fish schools). While IMs provide a rigorous framework for dimensionality reduction and stability analysis, their computation in discrete-time settings remains difficult. Traditional methods, such as local power series expansions (PSE), suffer from the "curse of dimensionality" regarding the combinatorial growth of coefficients and are limited by the radius of convergence, often failing far from the equilibrium point. Conversely, purely data-driven or physics-informed neural network (PINN) approaches, while scalable, may lack the interpretability and rapid local convergence properties of series expansions near equilibria.

Methodology
The authors propose a hybrid physics-informed machine learning framework that synergistically combines multivariate polynomial series and shallow neural networks (NNs) to approximate the functional relationship $x = \pi(y)$ defining the IM.

1. Mathematical Formulation: The problem is posed as solving a system of nonlinear functional equations (NFEs) derived from the invariance condition:
   $$\pi(Ay + g(y)) = B\pi(y) + Cy + f(\pi(y), y)$$
   where $y$ represents the exosystem state and $x$ the driven state.

2. Hybrid Architecture: The approximation $\tilde{\pi}_{HxS}(y)$ is constructed as a piecewise function:

   • Inside a hyperrectangle $D$ (centered at equilibrium): A polynomial series (Power, Legendre, or Chebyshev) is used to exploit geometric/exponential convergence properties near the equilibrium.
   • Outside $D$: A single-layer feedforward neural network (SLFNN) is employed to leverage universal approximation capabilities in regions where polynomial series may diverge or where the radius of convergence is exceeded.
   • Continuity: A Heaviside-based switching function ensures $C^0$ continuity at the boundary $\partial D$, enforced via a continuity penalty in the loss function.

3. Optimization: The learning task is formulated as a physics-informed optimization problem minimizing a composite loss function comprising:

   • NFE Residuals: Enforcing the invariance equation at collocation points.
   • Equilibrium Constraint: Ensuring the manifold passes through the equilibrium point.
   • Continuity Constraint: Matching polynomial and NN outputs at the boundary.

   The optimization is solved using the Levenberg–Marquardt (LM) algorithm. Crucially, the authors derive analytical expressions for all derivatives of the residuals with respect to the polynomial coefficients and NN parameters, arguing that this approach yields superior reliability and efficiency compared to finite differences or automatic differentiation, particularly for stiff systems.

4. Theoretical Foundations: The paper establishes a universal approximation theorem for the standalone NN component under specific assumptions (Barron regularity of the IM, Lipschitz continuity of the system dynamics, and bounded composition of the measure with respect to the exosystem). The theorem guarantees an $L^2$ convergence rate of $O(L^{-1/2})$ with respect to the network width $L$.

Key Contributions

• Hybrid Scheme: Introduction of a novel hybrid architecture that integrates the interpretability and local convergence of polynomial series with the global approximation power of neural networks for discrete-time IMs.
• Analytical Derivatives: Derivation of closed-form analytical derivatives for the hybrid scheme's residuals, facilitating robust LM optimization.
• Theoretical Guarantees: Establishment of a universal approximation theorem and convergence rates for the NN component applied to discrete-time invariance equations.
• Benchmarking: Comprehensive numerical assessment comparing the hybrid scheme against standalone PINNs and traditional PSEs.

Results
The framework was evaluated on two benchmark examples:

1. Enzymatic Bioreactor (1D driven, 1D exosystem): The hybrid scheme (using Power, Legendre, or Chebyshev series) achieved mean relative $L^2$ errors on the order of $10^{-5}$, significantly outperforming the standalone PSE (errors $\sim 10^{-3}$) and slightly outperforming the standalone PINN. The hybrid approach demonstrated high accuracy near the equilibrium (due to the polynomial component) and maintained accuracy far from it (due to the NN component).
2. Car-Following Model (20D driven, 2D exosystem): In this higher-dimensional case, the hybrid scheme again outperformed the PSE (errors $\sim 10^{-2}$) and showed slightly better global accuracy than the standalone PINN (errors $\sim 10^{-3}$). The hybrid scheme successfully generalized to the 20-dimensional manifold, maintaining homogeneous error distribution across the domain.

An additional validation case with a known analytic IM ($x = \ln(1+y)$) confirmed that while pure polynomial expansions fail outside their radius of convergence, the hybrid scheme maintains accuracy across the entire domain, with performance improving as the polynomial degree increases.

Significance and Claims
The paper claims that the proposed hybrid physics-informed scheme offers a comparative advantage over standalone polynomial expansions and standalone neural networks. Specifically:

• It overcomes the locality limitation of polynomial series, which fail outside their radius of convergence.
• It improves upon standalone PINNs by leveraging the rapid convergence of polynomials near the equilibrium, resulting in higher numerical accuracy in that region.
• It provides a one-step, plug-and-play solution that avoids the complexity of linearizing the problem or using multi-step procedures often required by traditional methods.

The authors modestly note that while the method is highly effective for low- to medium-dimensional manifolds (where polynomial complexity remains tractable), the combinatorial growth of coefficients remains a bottleneck for very high dimensions. They suggest that for such cases, a standalone PINN approximation may be more appropriate. The paper concludes by identifying future work directions, including the rigorous analysis of the hybrid scheme's convergence properties and the incorporation of parsimonious tensor-product bases.