🤖 machine learning

Enhancing Physics-Informed Neural Networks Through Feature Engineering

This paper introduces SAFE-NET, a single-layered architecture leveraging Fourier features and optimized conditioning to solve partial differential equations with significantly higher efficiency, lower error, and fewer parameters than existing Physics-Informed Neural Networks.

Original authors: Shaghayegh Fazliani, Zachary Frangella, Madeleine Udell

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shaghayegh Fazliani, Zachary Frangella, Madeleine Udell

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to predict how a wave moves across a pond, how heat spreads through a metal plate, or how air flows around a car wing. These are described by complex mathematical rules called Partial Differential Equations (PDEs).

For a long time, scientists have used Physics-Informed Neural Networks (PINNs) to solve these problems. Think of a standard PINN as a very smart, but very naive, student. It has a massive brain (a deep neural network with many layers) and is told the rules of physics. However, because it starts with no idea what the answer looks like, it has to guess and check millions of times. It's like trying to find a specific needle in a haystack by randomly poking the hay with a giant stick. It takes a long time, uses a lot of energy, and often gets stuck.

Recently, some researchers tried to help this student by giving it "feature engineering"—essentially handing it a map or a hint about what the answer might look like. But these previous maps were often rigid, hard to tune, or too expensive to use.

This paper introduces a new method called SAFE-NET. Here is how it works, using simple analogies:

1. The "Smart Shortcut" (Feature Engineering)

Instead of letting the computer guess everything from scratch, SAFE-NET gives the computer a head start. It uses Fourier features, which are like musical notes.

The Analogy: Imagine trying to recreate a complex song. A standard neural network tries to learn every single sound wave from scratch. SAFE-NET, however, is given a set of pre-tuned musical instruments (sine and cosine waves) that are known to be good at making music.
The Twist: Unlike previous methods that gave the student a fixed set of instruments (which might be out of tune for the specific song), SAFE-NET lets the student tune the instruments while playing. It learns exactly which "frequencies" (notes) are needed for the specific problem at hand.

2. The "One-Layer Genius"

Most modern AI models are like skyscrapers with dozens of floors (deep layers). SAFE-NET is a single-story house.

The Analogy: You don't need a 50-story building to store a few books; a small, well-organized shelf works better. SAFE-NET proves that for many physics problems, you don't need a massive, deep network. By using the right "features" (the tuned musical notes), a simple, single-layer network can solve the problem just as well, but much faster.
The Result: It uses 70 to 100 times fewer parameters (memory and brain cells) than the giant, complex models currently popular in the field.

3. The "Smooth Road" (Conditioning)

One of the biggest problems with training these AI models is that the "road" to the correct answer is full of potholes and steep cliffs (mathematically called an "ill-conditioned landscape"). This makes the optimizer (the driver) crash or get stuck.

The Analogy: Imagine driving a car to a destination. Standard methods are like driving on a rocky, bumpy dirt road where the car shakes violently. SAFE-NET acts like a paving crew that smooths out the road before you even start driving.
The Result: Because the road is smooth, the car (the optimizer) can drive straight to the destination without crashing. This allows the model to use powerful, fast-driving techniques (like L-BFGS) that usually fail on bumpy roads.

4. The "Cheat Sheet" (Domain Knowledge)

SAFE-NET also looks at the specific rules of the game before starting.

The Analogy: If you are solving a puzzle about a wave bouncing off a wall, SAFE-NET looks at the wall and says, "Okay, the wave must be zero at the wall." It builds this fact directly into its starting point. It doesn't just guess; it uses the known physics (like the shape of the container or the starting temperature) to build a better foundation.

What Did They Find?

The authors tested SAFE-NET on eight different types of physics problems (waves, heat, fluid flow, etc.).

Speed: It trained 30% faster in terms of time per step and reached the solution in fewer than 30% of the training steps compared to deep networks.
Efficiency: It uses 53% fewer parameters than other "feature engineering" methods and 70-100x fewer than the massive "state-of-the-art" models.
Accuracy: For smooth problems (like waves and heat), it was often the most accurate.
The Catch: It works best on "smooth" problems. If the problem involves sudden, violent shocks (like a sonic boom or a traffic jam crash), the simple musical notes struggle a bit more, though it still performed respectably.

The Big Takeaway

The paper challenges the idea that "bigger is always better" in AI. It suggests that for solving physics equations, smart, simple features combined with a well-tuned, shallow network are often superior to massive, deep, complex networks. It's not about having a bigger brain; it's about having the right tools and a smoother path to the answer.

Technical Summary: Enhancing Physics-Informed Neural Networks Through Feature Engineering

Problem Statement
Physics-Informed Neural Networks (PINNs) offer a mesh-free approach to solving Partial Differential Equations (PDEs) by minimizing the residual of the differential operator. However, PINNs face fundamental training challenges, primarily stemming from the ill-conditioning induced by differential operators in the residual loss. This ill-conditioning creates a poor optimization landscape, causing popular first-order optimizers like Adam to converge slowly and quasi-Newton methods like L-BFGS to diverge or stall. While recent trends in deep learning favor complex, multi-layer architectures (e.g., Transformers, large-scale operator learning models) to implicitly learn relevant features, these approaches often require millions of parameters and extensive computational resources. Conversely, existing feature engineering methods (e.g., Random Fourier Features, Radial Basis Functions) often rely on fixed or random priors, require sensitive hyperparameter tuning, and can be computationally expensive, limiting their generalizability and stability.

Methodology: SAFE-NET
The paper introduces SAFE-NET (Single-layered Adaptive Feature Engineering NETwork), a framework designed to improve PINN training through explicit, adaptive feature engineering rather than architectural depth. SAFE-NET combines three core components:

Adaptive Fourier Basis: Unlike standard Random Fourier Features (RFF) which use fixed, random frequencies, SAFE-NET employs a single hidden layer network where the frequencies ( $\omega_x, \lambda_t$ ) and amplitudes of the Fourier basis terms are trainable. The network utilizes a tensor product structure of sine and cosine functions to capture 2D (spatial-temporal) spectral properties. The frequencies are initialized based on harmonic analysis (e.g., $\ell\pi/k$ ) to align with the natural structure of PDE solutions, and the amplitudes are initialized to unity.
Domain Knowledge Features: The framework explicitly incorporates domain-specific features derived from initial conditions (ICs) and boundary conditions (BCs). If an IC or BC contains specific functional forms (e.g., trigonometric, polynomial, or exponential terms), these are extracted and concatenated with the Fourier features. For homogeneous Dirichlet conditions, features satisfying the boundary constraints (e.g., $\sin(n\pi x/L)$ ) are prioritized during initialization.
Normalization and Optimization: A critical design choice is the normalization of combined features (centered $L_2$ normalization) prior to network input. This stabilizes the optimization landscape, preventing divergence when using quasi-Newton optimizers like L-BFGS. The authors utilize a two-stage optimization schedule: an initial "warm-up" phase using Adam to escape saddle points, followed by L-BFGS for rapid local convergence. They also propose a restart strategy, $(Adam + L-BFGS)_2$ , where L-BFGS is restarted after stalling to further improve convergence on challenging PDEs.

Key Contributions

Efficient Architecture: SAFE-NET achieves high accuracy with a single hidden layer and significantly fewer parameters (average 53% fewer than competing feature engineering methods and 70–100 $\times$ fewer than state-of-the-art large-scale architectures like PINNsFormer or FNO).
Implicit Preconditioning: The paper demonstrates that SAFE-NET implicitly preconditions the loss landscape. Spectral density analysis of the Hessian matrix reveals that SAFE-NET shifts eigenvalues closer to 1, significantly reducing the condition number compared to standard PINNs and other baselines. This leads to a more favorable optimization landscape.
Robustness to Optimizers: By normalizing features and using adaptive frequencies, SAFE-NET is compatible with high-performance optimizers like L-BFGS, which often fail or diverge with other feature engineering methods due to ill-conditioning.
Empirical Superiority: Across eight diverse PDE benchmarks (including linear, nonlinear, time-dependent, and steady-state problems), SAFE-NET consistently outperforms or matches the performance of deeper networks and complex architectures while training in less than 30% of the epochs required by competitors.

Results
Experimental results on benchmarks such as the Wave, Heat, Diffusion, Convection, Burgers, Allen-Cahn, and Navier-Stokes equations show:

Accuracy: SAFE-NET achieves the best or comparable relative $L_2$ error (L2RE) on 5 out of 8 tested PDEs when using the $(Adam + L-BFGS)_2$ schedule. It significantly outperforms baselines on smooth PDEs (Wave, Heat, Diffusion).
Handling Discontinuities: While SAFE-NET is designed for smooth solutions, it still performs competitively on nonlinear problems with steep gradients (Allen-Cahn, Burgers), though it is noted that methods specifically designed for shocks (like RBF-P) may still hold an edge for highly discontinuous problems.
Efficiency: SAFE-NET is approximately 95% faster per epoch than competing feature engineering approaches. It converges to comparable accuracy with 70–100 $\times$ fewer parameters than large-scale operator learning models.
Spectral Analysis: Spectral density plots confirm that SAFE-NET reduces the magnitude and density of large eigenvalues in the Hessian matrix, validating the claim of improved conditioning.

Significance and Claims
The paper challenges the prevailing belief that modern deep learning must rely on massive, complex architectures to solve PDEs effectively. It argues that for PDEs with sufficiently smooth solutions, traditional techniques like feature engineering—when made adaptive and integrated with domain knowledge—can yield superior efficiency and accuracy. The authors claim that SAFE-NET highlights the importance of inductive bias derived from the mathematical structure of PDEs (Fourier analysis, boundary conditions) over the "black box" feature learning of deep networks. The work suggests that feature engineering provides a lasting, complementary value to deep learning in scientific computing, offering a path to high-accuracy solutions with minimal computational overhead, particularly for low-to-moderate dimensional problems. The paper modestly notes that extending these benefits to truly high-dimensional problems remains a future challenge.