Enhancing Physics-Informed Neural Networks with Domain-aware Fourier Features: Towards Improved Performance and Interpretable Results

This paper proposes Domain-aware Fourier Features (DaFFs) to enhance Physics-Informed Neural Networks (PINNs) by embedding domain-specific characteristics into positional encoding, which significantly improves training efficiency, accuracy, and interpretability while eliminating the need for explicit boundary condition losses and loss balancing schemes.

The Gist

Imagine you are trying to teach a robot to solve a complex puzzle, like predicting how a bridge will bend under heavy traffic or how sound waves bounce around a room. In the world of artificial intelligence, this is done using Physics-Informed Neural Networks (PINNs).

Think of a standard neural network as a very smart student who has never seen a physics textbook. If you ask it to solve a physics problem, it tries to guess the answer by looking at data points. But if you don't give it enough data (which is often the case in real-world engineering), it starts guessing wildly.

PINNs are like giving that student a physics textbook. You tell the computer, "Hey, you must follow the laws of physics (like Newton's laws or wave equations) while you learn." This is done by adding a "penalty" to the student's homework grade if their answer breaks the laws of physics.

However, the paper points out two big problems with this current approach:

1. The "Balancing Act" Nightmare: The student has to balance two things: getting the data right and following the physics rules. Often, the physics rules are so strict or so different from the data that the student gets confused, gets stuck, or takes forever to learn. It's like trying to juggle while riding a unicycle; the more rules you add, the harder it is to keep from falling.
2. The "Black Box" Mystery: Even when the student gets the right answer, we don't really know how they did it. We can't look inside their brain to see which part of the physics rule they used. This makes it hard to trust them in critical situations.

The Solution: "Domain-Aware Fourier Features" (DaFFs)

The authors propose a clever new way to teach the student, which they call Domain-Aware Fourier Features (DaFFs).

The Analogy: The Custom-Made Suit vs. The Random Fabric

• Old Way (Random Fourier Features): Imagine you are sewing a suit for a person, but you just grab random pieces of fabric from a giant bin. You hope that by mixing enough random pieces, you'll accidentally create a suit that fits perfectly. Sometimes it works, but often you get a weird, baggy suit, and you have to spend hours sewing and unsewing (tuning) to make it fit.
• The New Way (DaFFs): Instead of grabbing random fabric, you first measure the person's body (the "domain") and the shape of the room they live in (the "boundary conditions"). You then cut the fabric specifically to fit that person and that room perfectly from the very first stitch.

In technical terms, the authors use the mathematical "shape" of the problem (solving a specific equation called the Laplace operator) to create the input features for the neural network. Because these features are built to fit the boundaries of the problem perfectly, the student doesn't need to be punished for breaking the rules. The rules are baked into the fabric itself!

The Result:

• Faster Learning: The student doesn't waste time trying to figure out the boundaries. They jump straight to solving the main puzzle. The paper shows this method is orders of magnitude faster and more accurate than the old methods.
• No More Balancing: You don't need to juggle different "penalty weights" anymore. There is only one goal: solve the equation.

The Second Innovation: Making the "Black Box" Transparent

The second part of the paper is about Explainability. Even with the new "custom suit," we still want to know why the robot made a specific prediction.

The authors use a technique called LRP (Layer-wise Relevance Propagation).

The Analogy: The Detective's Flashlight

Imagine the neural network is a dark room, and the prediction is a lightbulb at the end. We want to know which wires (inputs) are powering that lightbulb.

• Old Models: When you shine the flashlight (LRP) on old models, the light is scattered everywhere. It looks like a mess of sparks. You can't tell which wire is actually important. It's like trying to find the source of a fire in a room full of random sparks.
• The New Model (DaFFs): When you shine the flashlight on the new model, the light is focused and clear. You can see exactly which "threads" of the physics are doing the heavy lifting. The model's reasoning is logical and matches what a human physicist would expect.

Why This Matters

This paper is a big step forward because it does two things at once:

1. It makes the math easier: By building the rules into the input, the computer learns faster and makes fewer mistakes.
2. It makes the AI trustworthy: By using the new "flashlight" technique, we can actually see why the AI thinks what it thinks. This is crucial for engineers and scientists who need to trust the computer before they build a bridge or design a medical device.

In a nutshell: The authors took a difficult, confusing way of teaching computers physics and replaced it with a method that builds the rules into the foundation of the learning process. The result is a faster, smarter, and more transparent AI that doesn't just guess the answer—it understands the logic behind it.

Technical Summary

1. Problem Statement

Physics-Informed Neural Networks (PINNs) are deep learning models designed to solve Partial Differential Equations (PDEs) by embedding physical laws directly into the loss function. Despite their potential, vanilla PINNs face significant challenges:

• Training Difficulties: They suffer from "gradient stiffness," where different components of the loss function (PDE residuals vs. boundary/initial conditions) converge at vastly different rates, requiring complex loss-balancing schemes.
• Spectral Bias: Standard fully connected networks tend to learn low-frequency solutions, struggling to capture high-frequency features inherent in many PDEs.
• Interpretability: PINNs are often treated as black boxes. When inputs are merely spatial or temporal coordinates, standard Explainable AI (XAI) methods struggle to provide meaningful insights because simple feature attribution scores lack physical context.
• Random Fourier Features (RFFs) Limitations: While RFFs help mitigate spectral bias, they introduce non-trainable randomness, do not inherently satisfy boundary conditions (requiring separate loss terms), and their stochastic nature hinders interpretability.

2. Methodology

The authors propose a novel modeling approach combining Domain-Aware Fourier Features (DaFFs) with a Layer-wise Relevance Propagation (LRP) explainability framework.

A. Domain-Aware Fourier Features (DaFFs)

Instead of using random frequencies (as in RFFs), DaFFs are derived analytically or numerically from the eigenvalue decomposition of the Laplace operator on the specific problem domain $\Omega$, subject to the boundary conditions.

• Mathematical Basis: The features $\phi_j(x)$ are eigenfunctions satisfying $-\nabla^2 \phi_j = \lambda_j \phi_j$ with boundary conditions $h(\phi_j) = 0$.
• Hard Constraints: Because these features are constructed to be zero at the boundaries (for homogeneous Dirichlet conditions) or satisfy specific derivative conditions, they inherently satisfy the boundary and initial conditions.
• Architecture Impact: This allows the removal of bias terms in the neural network layers. Consequently, the boundary condition loss terms ($L_b, L_c$) are eliminated from the objective function. The training reduces from a multi-objective problem to a single-objective problem (minimizing only PDE residuals $L_r$), removing the need for adaptive loss balancing.
• Flexibility: While derived for homogeneous Dirichlet conditions, the method extends to Neumann and Robin conditions via linear combinations of eigenfunctions or specific phase shifts.

B. Explainability Framework (LRP)

To address the interpretability gap, the authors adapt Layer-wise Relevance Propagation (LRP) for PINNs.

• Mechanism: LRP propagates the prediction relevance backward from the output to the input, adhering to a conservation principle ($\sum R_i = \sum R_j$).
• Adaptation: The authors modify standard LRP rules (specifically LRP-$\epsilon$) to handle residual skip connections and the specific behavior of DaFFs, where activations at boundary points are naturally zero.
• Goal: To extract relevance attribution scores that reveal which input features (spatial coordinates or Fourier components) drive the model's predictions, ensuring these attributions align with physical laws.

3. Key Contributions

1. Proposal of DaFFs: A new class of positional encoding that embeds domain geometry and boundary conditions directly into the input representation, eliminating the need for explicit boundary loss terms.
2. Simplified Optimization: By satisfying boundary conditions by design, the training process is simplified to a single loss term, significantly reducing computational cost and avoiding gradient stiffness issues associated with loss balancing.
3. XAI Framework for PINNs: Development of a tailored LRP framework that successfully extracts physically consistent relevance scores from PINNs, even when inputs are purely spatial coordinates.
4. Feature Selection Guidance: Demonstration that LRP can be used to identify the most relevant spectral components (eigenmodes) for a given problem, aiding in hyperparameter tuning.

4. Experimental Results

The authors evaluated their approach on two benchmark PDEs: the Kirchhoff-Love plate equation (4th order) and the Helmholtz equation (2nd order). They compared Vanilla PINNs, PINN-RFFs, and PINN-DaFFs.

• Accuracy: PINN-DaFFs achieved orders-of-magnitude lower errors compared to the other models.
  • Kirchhoff Example: Validation loss of $6.42 \times 10^{-18}$ for DaFFs vs. $3.01 \times 10^{-7}$ for RFFs.
  • Helmholtz Example: Validation loss of $2.32 \times 10^{-11}$ for DaFFs vs. $7.75 \times 10^{-6}$ for RFFs.
• Efficiency: PINN-DaFFs converged significantly faster (fewer epochs) and required less training time (e.g., ~31 mins vs. ~2h 51m for Helmholtz) due to the single-objective loss function.
• Interpretability Analysis:
  • Vanilla PINNs: Showed scattered and inconsistent relevance scores across the domain, indicating poor physical consistency.
  • PINN-RFFs: Showed contributions scattered across random spectral components. The model did not consistently converge to the domain's representative spectral modes, suggesting it fell into local optima.
  • PINN-DaFFs: Produced physically consistent attributions. The relevance scores aligned with the expected eigenmodes of the problem (e.g., specific $m, n$ harmonic indices), confirming the model learned the correct underlying physics.

5. Significance

This work bridges the gap between high-performance scientific machine learning and interpretability.

• Robustness: By removing the reliance on loss balancing and random feature selection, DaFFs provide a more robust and reproducible training pipeline for PDEs.
• Trustworthiness: The integration of LRP demonstrates that PINNs can be made transparent. The fact that DaFFs yield physically consistent relevance scores validates that the model is not just "fitting" the data but is learning the actual dynamics of the system.
• Future Directions: The approach lays the groundwork for solving more complex problems with non-homogeneous boundary conditions and sparse data, while using XAI to guide feature selection and model architecture design.

In summary, the paper demonstrates that Domain-Aware Fourier Features not only solve the training instability and accuracy issues of standard PINNs but also transform them into interpretable models where the learned features directly correspond to the physical eigenmodes of the system.