🤖 machine learning

Do physics-informed neural networks (PINNs) need to be deep? Shallow PINNs using the Levenberg-Marquardt algorithm

This paper demonstrates that shallow physics-informed neural networks (PINNs) can efficiently solve forward and inverse nonlinear PDE problems by employing the Levenberg-Marquardt algorithm and analytical derivatives, outperforming standard optimization methods like BFGS in speed and accuracy.

Original authors: Muhammad Luthfi Shahab, Imam Mukhlash, Hadi Susanto

Published 2026-02-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Muhammad Luthfi Shahab, Imam Mukhlash, Hadi Susanto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Do You Need a Supercomputer to Solve Nature’s Riddles?

Imagine you are trying to solve a massive, complex jigsaw puzzle of a landscape.

Most scientists today use "Deep" Neural Networks to solve physics problems (like how heat moves through a metal rod or how waves crash in the ocean). Using a "Deep" network is like hiring a massive team of 1,000 specialists, each sitting in a different room, passing notes to one another to figure out where a single puzzle piece goes. It’s incredibly powerful, but it’s slow, expensive, and requires a massive amount of "brainpower" (computing energy and high-end GPUs).

This paper asks a provocative question: "Do we really need that massive team? Could a small, elite squad of experts do the same job faster and better?"

The authors suggest that instead of a "Deep" team, we can use a "Shallow" network—a much smaller, leaner group—as long as we give them a much better "Management Strategy."

The Two Main Characters

To understand the paper, you need to meet the two "workers" being compared:

1. The "Deep" Worker (The Standard Way)

Think of this as a massive, sprawling corporation. They have many layers of middle management. When they make a mistake, the information has to travel through dozens of levels to get back to the top. It takes a long time to correct errors, and they often get "lost in the bureaucracy" (this is what scientists call training inefficiency).

2. The "Shallow" Worker + The "LM" Manager (The Paper’s Way)

The authors propose using a "Shallow" network. This is like a small startup with only two levels of management. It’s much faster and uses less "office space" (memory).

But here is the secret sauce: To make sure this small team doesn't make mistakes, they hired a world-class manager called the Levenberg–Marquardt (LM) algorithm.

The Old Manager (BFGS): This manager is like a person walking down a foggy mountain. They feel the slope under their feet and take a step downward. It works, but they might wander around aimscessly for a long time before finding the bottom.
The New Manager (LM): This manager is like someone with a high-tech GPS and a drone. Instead of just feeling the slope, they look at the curvature of the mountain. They don't just ask, "Which way is down?" they ask, "How fast is the slope changing, and how can I leap toward the bottom most efficiently?"

What did they actually do? (The Experiment)

The researchers put these two methods to the test using famous "boss battles" from the world of physics (equations like Burgers, Schrödinger, and Bratu). These are mathematical puzzles that describe how fluids flow or how quantum particles behave.

The Results were stunning:

Accuracy: The "Small Startup" (Shallow PINN) with the "High-Tech Manager" (LM) actually found the answers more accurately than the "Massive Corporation" (Deep PINNs).
Speed: The small team reached the answer much faster.
Efficiency: In one test, the small team used 25 times fewer parameters (less "brainpower") but still beat the heavyweights.

Why does this matter to you?

Right now, solving complex physics problems requires incredibly expensive supercomputers and massive amounts of electricity. This paper proves that complexity isn't always about size; it's about intelligence.

By using "Shallow" networks paired with "Smart" math (the LM algorithm), we can:

Run simulations on a regular laptop instead of a multi-million dollar supercomputer.
Solve problems faster, which could lead to quicker breakthroughs in weather forecasting, engineering, or medicine.
Save energy, making AI and physics simulations much more "green."

The takeaway: You don't need a giant, heavy machine to solve a difficult problem; you just need a lean, smart team and a very good map.

Technical Summary: Shallow Physics-Informed Neural Networks (PINNs) using the Levenberg–Marquardt Algorithm

1. Problem Statement

The paper addresses a fundamental question in the field of scientific machine learning: Do Physics-Informed Neural Networks (PINNs) necessarily need to be deep to be effective?

While the standard PINN framework (popularized by Raissi et al.) typically utilizes deep architectures and first-order or quasi-Newton optimizers (like Adam or L-BFGS), these deep models often suffer from high computational costs and training inefficiencies. The authors investigate whether shallow neural networks (specifically those with only two hidden layers), when paired with a more robust second-order optimization strategy, can match or exceed the accuracy and efficiency of deep PINNs for both forward problems (solving PDEs) and inverse problems (identifying unknown PDE parameters).

2. Methodology

The authors propose a paradigm shift in how PINNs are trained, moving from a standard minimization of a scalar loss function to the solution of a nonlinear system of equations.

Architecture: They employ shallow feed-forward neural networks denoted as $NN(a, m_1, m_2, 1)$ , where $a$ is the number of inputs, $m_1$ and $m_2$ are the number of neurons in the two hidden layers, and the output is a single scalar.
Constraint Enforcement: To improve stability, initial and boundary conditions are enforced directly within the network's functional form (using auxiliary functions $p(x, t)$ and $q(x)$ ), rather than solely through penalty terms in the loss function.
Optimization (Levenberg–Marquardt): Instead of using gradient descent, the authors reformulate the PDE residual as a nonlinear least-squares problem: $F(W) = 0$ . They employ the Levenberg–Marquardt (LM) algorithm, which adaptively interpolates between the Gauss-Newton algorithm and the method of gradient descent.
Analytical Differentiation: A core technical contribution is the derivation of exact analytical expressions for the neural network's derivatives (up to the second order) with respect to both input variables ( $x, t$ ) and network parameters ( $W$ ). This allows for the explicit construction of the Jacobian matrix ( $J$ ), which is required for the LM update step:
$\Delta W = (J^T J + \kappa I)^{-1} J^T F$
This avoids the "black box" nature of automatic differentiation and provides a transparent view of how weights influence the differential operators.

3. Key Contributions

Reformulation of PINNs: Transitioning the PINN training process from a standard optimization problem to a nonlinear system-solving problem.
Analytical Jacobian Framework: Providing a complete, closed-form mathematical derivation for the Jacobian in PINNs, which clarifies the dependency structure between shared weights and the resulting derivatives.
Efficiency via Shallow Architectures: Demonstrating that reducing network depth significantly lowers the computational cost of Jacobian evaluation without sacrificing (and often improving) accuracy.
Comparative Benchmarking: Establishing a rigorous comparison between the LM algorithm and the widely used BFGS/L-BFGS methods.

4. Results

The methodology was tested on four complex nonlinear PDEs: the Burgers, nonlinear Schrödinger, Allen–Cahn, and 3D Bratu equations.

Forward Problems: In the Burgers and Schrödinger equations, the shallow LM-based PINN achieved relative $L_2$ errors several orders of magnitude lower than the BFGS method. Notably, the shallow network (e.g., 751 parameters) outperformed the original deep PINN (3021 parameters) in accuracy.
Inverse Problems: For the Allen–Cahn and 3D Bratu equations, the LM algorithm identified unknown PDE parameters ( $\lambda$ ) with much higher precision and faster convergence than BFGS.
Computational Performance: While a single LM iteration is more expensive than a BFGS iteration, the LM approach requires significantly fewer iterations to reach convergence, resulting in lower total computational time for high-accuracy solutions.

5. Significance

This work challenges the "deeper is better" assumption in PINNs. It demonstrates that optimization strategy is as critical as architecture depth. By using shallow networks and second-order methods like LM, researchers can achieve high-fidelity solutions for complex PDEs using standard CPU hardware, making PINNs more accessible to those without high-performance GPU clusters. Furthermore, the analytical framework provided offers a theoretical foundation for understanding the internal mechanics of physics-informed learning.