Physics Informed Neural Network using Finite Difference Method

1. Problem Statement

Physics-Informed Neural Networks (PINNs) have emerged as a powerful tool for solving partial differential equations (PDEs) by embedding physical laws directly into the neural network's loss function. Traditionally, PINNs rely on Automatic Differentiation (AD) to compute the partial derivatives required for the physics-informed loss term ($L_f$).

However, the authors identify several limitations with AD-based PINNs:

• Computational Cost: AD is computationally expensive, often requiring significant time and GPU resources, especially for complex PDEs.
• Convergence Issues: AD-based PINNs sometimes struggle to converge theoretically and may exhibit lower accuracy when the number of collocation points is small.
• Black-Box Nature: AD acts as a "black box," making it difficult to decouple the PDE solver from the neural network training process.

The paper addresses the need for a more efficient, lightweight alternative to AD that maintains or improves accuracy, particularly in scenarios with limited data (low collocation points).

2. Methodology: FDM-PINN

The authors propose FDM-PINN, a novel approach that replaces Automatic Differentiation with Finite Difference Method (FDM) for estimating the PDE loss functions.

Core Concept

Instead of using AD to compute gradients ($\frac{\partial u}{\partial x}$, etc.) through the computational graph, the authors:

1. Discretize the Domain: They map the continuous PDE domain onto a discrete grid of nodes.
2. Approximate Derivatives: They use standard FDM stencils (e.g., central differences) to approximate the derivatives of the neural network output ($u$ or $V$) at these grid points.
3. Decouple the Solver: The PDE solver is effectively separated from the backpropagation of the neural network weights. The FDM calculation acts as a fixed operator (like a Laplacian filter in image processing) applied to the network output.

Mathematical Formulation

The total loss function remains the sum of the data-driven Mean Square Error ($L_\mu$) and the physics-informed loss ($L_f$):
$$L_{PINN} = L_\mu + \lambda L_f$$

• Physics-Informed Loss ($L_f$): Instead of calculating $f(u)$ via AD, the authors approximate the PDE residual $f(u)$ using FDM.

  • For Laplace's Equation ($\nabla^2 V = 0$), the residual is approximated as the difference between a node's value and the average of its neighbors (identical to the SOR algorithm's residual step).
  • For Burgers' Equation ($u_t + uu_x - \nu u_{xx} = 0$), first and second-order derivatives are approximated using forward/backward difference stencils.

• Weight Update: The gradient of the loss with respect to the weights is computed using the chain rule:
  $$\nabla L_f = \frac{\partial L_f}{\partial \Gamma} \times \frac{\partial \Gamma}{\partial V}$$
  Where $\Gamma$ is the FDM approximation of the PDE residual. The authors assume $\frac{\partial \Gamma}{\partial V}$ (the gradient of the local surface) is a constant to simplify the backpropagation, avoiding the need for AD.

3. Key Contributions

• AD-Free Training: The paper demonstrates a fully functional PINN architecture that does not require Automatic Differentiation libraries (e.g., tf.gradients or dlgradient) to compute the physics loss.
• Low Implementation Cost: The method simplifies implementation by utilizing standard numerical differentiation, which is computationally cheaper than building complex AD graphs.
• Performance at Low Data: The approach is specifically optimized to perform well with fewer collocation points, a known weakness of standard AD-PINNs.
• Hardware Independence: The method can be executed efficiently on CPUs without requiring GPUs, unlike many AD-based implementations.

4. Experimental Results

The authors evaluated FDM-PINN on two benchmark problems: Laplace's Equation (rectangular conducting trough) and Burgers' Equation (shock hydrodynamics). They compared three models:

1. NN: Standard Neural Network (MSE loss only).
2. AD-PINN: Standard PINN using Automatic Differentiation.
3. FDM-PINN: The proposed method using Finite Differences.

Key Findings:

• Accuracy vs. NN: FDM-PINN significantly outperformed standard NNs in both equations, proving the value of physics constraints even without AD.
• Accuracy vs. AD-PINN:
  • Burgers' Equation: FDM-PINN was on par or slightly better than AD-PINN across various training sample sizes ($N_{ini}$ and $N_{bc}$).
  • Laplace's Equation: FDM-PINN consistently outperformed both NN and AD-PINN, showing lower $L_2$ errors, particularly in the range of 20–100 initial samples.
• Computational Efficiency:
  • Execution Time: FDM-PINN was significantly faster than AD-PINN.
    • For Burgers' equation, AD-PINN took ~5x longer than FDM-PINN (even when AD-PINN used a GPU and FDM-PINN used a CPU).
    • For Laplace's equation, FDM-PINN was slightly faster than AD-PINN.
  • Scalability: The method showed robustness with lower collocation points ($N_f$), where AD-PINN performance often degraded.

5. Significance and Future Work

Significance:
This work challenges the dominance of Automatic Differentiation in the PINN community. It proves that classical numerical methods (FDM) can be effectively integrated into deep learning frameworks to solve PDEs. This is particularly significant for:

• Resource-Constrained Environments: Applications where GPU access is limited or expensive.
• Simplicity: Reducing the complexity of the training pipeline by removing the dependency on complex AD graphs.
• Interpretability: The FDM approach offers a more transparent view of how the PDE residual is calculated compared to the "black box" nature of AD.

Limitations & Future Directions:

• Grid Dependency: The current method requires collocation points to be defined on a structured grid, limiting flexibility compared to mesh-free AD methods.
• Gradient Approximation: The assumption that $\frac{\partial \Gamma}{\partial V}$ is constant is a simplification; future work aims to compute this gradient more accurately.
• Extensions: The authors suggest extending the architecture to 2D Convolutional Neural Networks (CNNs) and exploring Markov Random Fields for the PDE solver to handle more complex, non-homogeneous outputs.

In conclusion, the paper establishes FDM-PINN as a viable, efficient, and often superior alternative to AD-based PINNs, particularly for problems requiring high efficiency or operating under data-scarce conditions.