Technical Summary: Curvature-aware Dynamic Precision Approach for Physics-Informed Neural Networks

1. Problem Statement

Physics-Informed Neural Networks (PINNs) have emerged as a powerful framework for solving partial differential equations (PDEs) by embedding physical laws directly into the training loss. However, PINN optimization is notoriously sensitive to numerical precision. Existing implementations face a critical trade-off:

• Single Precision (FP32): Computationally efficient but prone to "failure modes" where the loss appears to converge while the predicted solution remains far from the true PDE solution. This is often due to ill-conditioned optimization landscapes and the amplification of numerical errors by high-order derivative terms.
• Double Precision (FP64): Robust and capable of mitigating precision-related failures, but computationally expensive, especially on modern hardware optimized for lower-precision arithmetic.

Current adaptive strategies, such as mixed-precision training or adaptive collocation, often rely on static allocation or layer-wise precision changes. They do not explicitly adapt precision to the specific phases of training dynamics or leverage optimization geometry (curvature) to schedule precision switches. The core challenge is to reduce the computational cost of FP64 training while retaining its accuracy, without sacrificing the robustness required for complex PDEs.

2. Methodology

The authors propose a curvature-aware dynamic precision controller that adaptively switches between FP32 and FP64 during the training of PINNs. The method is built upon the following components:

2.1. Curvature Signal Extraction

The approach reuses curvature information already computed by the L-BFGS (Limited-memory Broyden–Fletcher–Goldfarb–Shanno) optimizer, a second-order quasi-Newton method commonly used for PINNs.

• L-BFGS maintains a history of parameter updates ($s_k$) and gradient updates ($y_k$).
• The authors define a directional curvature proxy, $\kappa_k = \frac{s_k^\top y_k}{s_k^\top s_k}$, which represents the Rayleigh quotient of the local Hessian approximation.
• A conditioning proxy, $\hat{\kappa}_{cond}$, is calculated as the ratio of the maximum to minimum curvature values within the current L-BFGS memory window. This serves as a lightweight indicator of the local optimization landscape's conditioning.

2.2. Precision Controller Logic

To stabilize the signal against short-term fluctuations, the conditioning proxy undergoes a log-transform and exponential moving average smoothing ($\tilde{z}_j$). The controller evaluates the smoothed signal and its slope ($\Delta \tilde{z}_j$) every 10 L-BFGS steps to decide the precision state:

• Switch to FP32: Occurs when the smoothed curvature proxy $\tilde{z}_j$ is below a problem-dependent threshold ($\tau_z$) and the slope is small, indicating a stable, well-conditioned phase.
• Switch to FP64: Occurs when $\tilde{z}_j$ exceeds the threshold, the slope increases rapidly, or parameter updates in FP32 become vanishingly small (indicating stagnation or numerical sensitivity).

This architecture-agnostic controller allows the training to utilize the speed of FP32 during stable phases and the robustness of FP64 during numerically critical phases (e.g., high curvature or ill-conditioning).

3. Key Contributions

1. Dynamic Precision Strategy: A novel approach that treats numerical precision as a dynamic variable adapted to training phases, rather than a fixed implementation choice.
2. Curvature-Aware Control: The first method to explicitly reuse L-BFGS curvature history as an online signal for precision scheduling in PINNs, linking optimization geometry directly to numerical stability.
3. Architecture Agnosticism: The method is demonstrated to be effective across various neural network architectures, including standard MLPs, PINNsFormer, PINNMamba, and KAN (Kolmogorov-Arnold Networks).
4. Cost-Accuracy Balance: The approach achieves FP64-level accuracy with reduced wall-clock time, effectively resolving the efficiency-accuracy trade-off in PINN training.

4. Experimental Results

The method was evaluated on four canonical PINN failure-mode benchmarks (Convection, Reaction, Wave, and Allen–Cahn equations) and an irradiance-driven plant-growth ODE.

• Performance vs. Fixed Precision:

  • Accuracy: The dynamic approach consistently matched or slightly exceeded the accuracy of full FP64 training (measured by rRMSE and rMAE). In contrast, pure FP32 training often failed to converge to acceptable solutions, producing errors one to two orders of magnitude larger.
  • Speed: The dynamic method reduced training time relative to full FP64 across all benchmarks. For example, on the Reaction equation, it achieved a 1.07x speedup over FP64 while maintaining lower error. On the Wave equation, it achieved a 1.04x speedup.
  • Trade-off: The method consistently improved the speed-accuracy trade-off, positioning results closer to the Pareto frontier compared to fixed-precision baselines.

• Architecture Effects:

  • While advanced architectures (PINNsFormer, PINNMamba, KAN) showed promise, they exhibited instability across random seeds for certain equations (e.g., Convection and Allen–Cahn).
  • The proposed dynamic precision controller applied to a standard vanilla MLP achieved robust convergence (100% convergence rate) across all four PDEs, outperforming the convergence rates of advanced architectures trained in fixed precision.
  • Interestingly, combining the dynamic controller with advanced architectures (like PINNsFormer) yielded further improvements, suggesting that smoother curvature trajectories in these architectures allow the controller to operate more efficiently.

• Phase-Dependent Sensitivity:

  • Analysis of the curvature signal revealed that numerical sensitivity is phase-dependent. Training phases with low, smooth curvature could be handled efficiently in FP32, while phases with high curvature or rapid signal changes required FP64.
  • The dynamic approach successfully navigated these phases, switching to FP64 only when necessary to prevent stagnation or instability.

5. Significance and Claims

The paper claims that the proposed curvature-aware dynamic precision approach offers a practical solution to the computational bottleneck of high-precision PINN training. By demonstrating that precision sensitivity in PINN optimization is phase-dependent, the authors argue that selectively applying higher precision only during numerically critical stages can lower computational costs without sacrificing predictive accuracy.

The authors position this work as a complementary advancement to architectural improvements. They note that while specific architectures can smooth the optimization landscape, they do not eliminate precision-related instability entirely. Therefore, dynamic precision control serves as a robust, architecture-agnostic layer that enhances the efficiency and reliability of PINN training across diverse problem settings, from standard PDE benchmarks to real-world ODE applications.

The study concludes that the method preserves FP64-level accuracy at a lower cost, making it a viable strategy for scaling PINNs to more complex scientific computing tasks where double precision is traditionally required for stability.