Technical Summary: AdamFLIP

Problem Statement

Physics-Informed Neural Networks (PINNs) offer a flexible framework for solving forward and inverse problems governed by partial differential equations (PDEs). However, standard PINN training typically relies on soft-penalty formulations, where the PDE residuals, initial conditions (ICs), and boundary conditions (BCs) are combined into a composite loss function using manually tuned weights. This approach suffers from several critical issues:

1. Ill-conditioning and Sensitivity: The performance is highly sensitive to the choice of penalty weights. Improper weighting can lead to slow convergence or solutions that fit the data while violating governing equations.
2. Scale Mismatch: Different loss terms often operate on different scales, meaning minimizing the composite loss does not guarantee the individual minimization of each term.
3. Constraint Satisfaction: Soft penalties often result in poor satisfaction of hard physical constraints (ICs/BCs), even when the overall loss is low.

While recent methods have attempted to incorporate constrained optimization (e.g., Augmented Lagrangian, Sequential Quadratic Programming) or hard-constraint architectures, many remain dependent on penalty schedules, require complex subproblem solving, or lack the scalability of adaptive optimizers like Adam.

Methodology: AdamFLIP

The authors propose AdamFLIP (Adaptive Momentum Feedback Linearization Optimization for Hard Constrained PINN), which reformulates PINN training as an equality-constrained optimization problem.

1. Problem Reformulation

Instead of minimizing a weighted sum of losses, the method separates the objective from the constraints:

• Forward Problems: The objective is the physics loss ($L_{phy}$), while IC and BC losses are treated as equality constraints ($h(\theta) = 0$).
• Inverse Problems: The objective is the data loss ($L_{data}$), while physics, IC, and BC losses are treated as constraints.
  This formulation explicitly separates feasibility (satisfying physical laws) from optimality (fitting data or minimizing residuals).

2. Feedback Linearization (FL)

Inspired by control theory, the method views the constraint residuals $y_t = h(\theta_t)$ as the output of a controlled dynamical system. The goal is to design a feedback input (the Lagrange multiplier $\lambda_t$) that drives these residuals toward stable, linear contraction dynamics:
$$y_{t+1} - y_t = -\eta K y_t$$
where $K$ is a gain matrix. By solving for $\lambda_t$ using the Jacobian of the constraints ($J_h$), the method enforces a controlled contraction of constraint violations without relying on iterative penalty updates.

3. Integration with Adaptive Momentum

Standard FL updates use a single global stepsize, which is suboptimal for neural networks where gradients are often anisotropic. AdamFLIP integrates the FL update with Adam-style adaptive moment estimation:

• It computes the feedback-linearized Lagrangian gradient: $g_t = \nabla f(\theta_t) + J_h(\theta_t)^\top \lambda_t$.
• It maintains first-moment ($m_t$) and second-moment ($v_t$) estimates of this gradient.
• The parameter update applies coordinate-wise adaptive scaling to the momentum direction, allowing for robust convergence in high-dimensional, non-convex settings.

4. Theoretical Guarantees

The paper provides a finite-time convergence analysis for a metric-compatible variant of AdamFLIP (Algorithm 2). Under standard smoothness and regularity assumptions, the authors prove that the best-iterate convergence rate for the combined stationarity residual and constraint violation is:
$$O\left(\frac{\log T}{\sqrt{T}}\right)$$
The authors note that proving convergence for the original Algorithm 1 (which uses a metric mismatch between the multiplier calculation and the update) remains an open problem due to the complexity of the adaptive metric.

Key Contributions

1. Formulation: A principled reformulation of PINN training as an equality-constrained problem, separating feasibility from optimality.
2. Algorithm: The introduction of AdamFLIP, which combines feedback linearization (for principled constraint handling) with Adam-style adaptive moments (for scalability and robustness).
3. Performance: Empirical demonstration that AdamFLIP consistently outperforms standard soft-penalty PINNs and state-of-the-art constrained optimizers (AL-PINN, trSQP-PINN) across various PDE benchmarks.
4. Theory: A finite-time convergence guarantee for a modified variant of the algorithm, establishing best-iterate convergence rates.

Experimental Results

AdamFLIP was evaluated on four benchmark PDEs (1D Burgers, 1D Time-Fractional Mixed Diffusion-Wave, 2D Heat, and 2D Navier–Stokes) in both forward and inverse settings.

• Accuracy: AdamFLIP consistently achieved the lowest relative $L_2$ errors. For the Navier–Stokes equations, it reduced the relative $L_2$ error by more than two-thirds compared to the next best method.
• Constraint Satisfaction: AdamFLIP significantly reduced errors in initial and boundary conditions (often by orders of magnitude) compared to standard PINNs and other constrained methods.
• Inverse Problems: In parameter identification tasks (e.g., recovering viscosity or fractional orders), AdamFLIP demonstrated superior accuracy and robustness, recovering parameters with significantly lower absolute errors than baselines.
• Computational Cost: AdamFLIP incurred only a modest overhead (typically $\le 30\%$) compared to standard PINN training, whereas methods like trSQP-PINN required 2–5 times longer due to subproblem solving.

Significance and Claims

The paper claims that AdamFLIP offers a simple yet powerful alternative to manual loss reweighting and classical constrained-training schemes. By embedding feedback-linearization principles into modern adaptive optimizers, the method:

• Eliminates the need for manual weight tuning, addressing a major bottleneck in PINN training.
• Ensures rigorous constraint satisfaction while maintaining the computational efficiency and scalability of first-order methods.
• Provides a unified framework that handles both forward and inverse problems effectively, even in challenging scenarios like fractional-order PDEs and coupled nonlinear systems (Navier–Stokes).

The authors position AdamFLIP not as a replacement for all existing methods, but as a robust, scalable optimizer that bridges the gap between control-theoretic constraint enforcement and deep learning optimization. They acknowledge limitations, such as the theoretical gap in proving convergence for the exact algorithm used in experiments and the assumption of full row rank for the constraint Jacobian, which may not hold near exact constraint satisfaction.