🤖 machine learning

ProPINN: Demystifying Propagation Failures in Physics-Informed Neural Networks

This paper identifies the root cause of PINN propagation failures as low gradient correlation between nearby points due to single-point processing, and proposes ProPINN, a new architecture that unifies these gradients to significantly outperform existing models.

Original authors: Yuezhou Ma, Haixu Wu, Hang Zhou, Huikun Weng, Jianmin Wang, Mingsheng Long

Published 2026-08-11

📖 9 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuezhou Ma, Haixu Wu, Hang Zhou, Huikun Weng, Jianmin Wang, Mingsheng Long

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to predict how a drop of ink spreads through a glass of water, or how heat travels through a metal rod. In the world of science and engineering, these movements are governed by complex mathematical rules called "physics equations." For decades, scientists have used powerful computers to solve these equations, but it's often like trying to untangle a giant knot of headphones: slow, difficult, and prone to getting stuck. Recently, a new kind of "smart robot" called a Neural Network has been introduced to solve these puzzles. These networks are like super-smart students that learn by looking at examples. However, when these students try to learn physics, they often hit a weird wall. They can memorize the rules for the starting point (like the ink drop at time zero) or the edges of the glass, but they completely fail to figure out what happens in the middle. It's as if the student knows the answer to the first question on a test but forgets everything in between, leaving the middle of the page blank or filled with nonsense. This frustrating phenomenon is what scientists call "propagation failure."

A new paper titled "ProPINN: Demystifying Propagation Failures in Physics-Informed Neural Networks" dives deep into why this happens and offers a clever fix. The authors, researchers from Tsinghua University, realized that the problem wasn't just that the robot was "forgetting" things; it was how the robot was built. They discovered that the standard design treats every single point in the simulation as an isolated island. Imagine a classroom where every student is working on their own piece of a giant puzzle, but they are forbidden from talking to their neighbors. Even if the student at the edge knows the picture, they can't tell the student in the middle what to do because there's no connection. The paper proves that this lack of "neighborly chat" causes the learning to fail. To fix this, they invented a new architecture called ProPINN. Instead of treating points as islands, ProPINN groups them into neighborhoods and forces them to share their "thoughts" (mathematically, their gradients) before making a decision. This simple change acts like a super-highway for information, allowing the correct solution to flow smoothly from the edges to the center. In tests, this new method didn't just fix the failures; it solved complex physics problems with 46% better accuracy than the previous best methods, all while running faster and using less computer power.

The Story of the Stuck Student

To understand the magic of ProPINN, we first need to look at the "stuck student" problem. In the world of solving physics equations, there are two main ways to teach a computer. The old way, used for years, is like drawing a grid on a map and calculating the answer for every single square one by one. It's reliable but slow. The new way uses Physics-Informed Neural Networks (PINNs). Think of a PINN as a neural network—a type of AI that learns patterns—trained not just on data, but on the laws of physics themselves. You tell the AI, "You must follow these rules," and it tries to guess the solution.

The problem arises because the AI is only directly told the correct answers at the very beginning (initial conditions) and the very edges (boundary conditions). For the rest of the space—the vast interior of the domain—the AI has to figure out the answer by itself, using only the physics rules. The paper argues that in the standard setup, the AI treats every single point in that interior space as if it were a separate, isolated student. When the AI updates its "brain" (its parameters) based on one point, it doesn't really help its neighbors. It's like if you learned a math trick for the number 5, but that knowledge didn't help you understand the number 6. Because the points are so isolated, the "correct" information from the edges never makes it to the middle. The AI gets stuck in a local trap, often guessing that the answer is zero everywhere because that's the easiest way to satisfy the rules, even though it's wrong.

The "Stiffness" of the Connection

The authors of the paper decided to investigate why this isolation happens. They looked at an older, trusted method called the Finite Element Method (FEM). In FEM, the computer breaks the problem into a mesh of connected triangles or squares. The key difference is that in FEM, the value at one point is mathematically tied to its neighbors. If you push one point, the neighbors feel it immediately. It's like a connected web of springs; pull one, and the whole web wiggles.

The researchers asked: "Why don't Neural Networks have this spring-like connection?" They proved that in standard PINNs, the connection between nearby points is incredibly weak. They introduced a new way to measure this connection called gradient correlation. Imagine the "gradient" as the direction the AI wants to move its brain to get a better answer. If two neighbors have gradients that point in the same direction, they are "correlated" and working together. If they point in opposite directions, they are fighting each other.

The paper proves a surprising fact: in standard PINNs, the gradients of nearby points often point in completely different directions, almost as if they are orthogonal (at a 90-degree angle) to each other. This means the AI is trying to learn two conflicting things at once for points that are right next to each other. This lack of agreement is the root cause of the "propagation failure." The information from the correct boundary gets blocked because the path through the middle is a chaotic mess of conflicting signals.

The ProPINN Solution: The Neighborhood Watch

So, how do you fix a student who won't talk to their neighbors? You don't just tell them to talk harder; you change the classroom layout. The authors propose ProPINN, a new architecture designed to force these neighbors to collaborate.

Here is how ProPINN works, using a playful analogy:
Imagine the standard PINN is a group of people standing in a field, each holding a piece of paper. They are told to solve a puzzle, but they can only look at their own paper. They are confused and stuck.
ProPINN changes the rules. Before they try to solve the puzzle, it gives each person a "neighborhood kit."

Differential Perturbation: The AI takes a single point and creates a few "ghost" versions of it, slightly shifted in different directions (like looking at the same spot from slightly different angles).
Shared Projection: All these points (the original and the ghosts) are fed into a shared "translator" that converts them into a common language.
Multi-Region Mixing: This is the magic step. The AI takes the information from the original point and its ghosts, mixes them together, and averages them out. It's like the student in the middle of the room now gets to hear the thoughts of everyone in their immediate circle before making a decision.

By mixing the gradients (the "thoughts") of nearby points, ProPINN ensures that the AI doesn't just learn for one point in isolation. It learns a consistent pattern for a whole region. This creates a strong "stiffness" or connection between neighbors, just like the springs in the old FEM method. The "correct" supervision from the boundaries can now flow smoothly through the mixed gradients, reaching the interior without getting lost.

The Results: Faster, Smarter, and More Accurate

The paper doesn't just talk about theory; they put ProPINN to the test against some of the toughest physics problems. They tested it on things like:

Convection: Simulating how a wave of heat or fluid moves quickly across a space.
Reaction-Diffusion: Modeling how chemicals mix and react, creating complex patterns.
Fluid Dynamics: Simulating the swirling, chaotic movement of air around a cylinder (the famous "Karman Vortex Street") and the flow of water in a torus (a donut shape).

In these tests, standard PINNs often failed miserably, producing errors that were huge or getting stuck in wrong solutions. Even advanced models that tried to use "attention mechanisms" (a fancy way of letting points talk to each other, similar to how Transformers work in language models) struggled. They were too slow and sometimes unstable.

ProPINN, however, shined.

Accuracy: It reduced the error by an average of 46% compared to the second-best model. In some specific tasks, like the 1D-Wave equation, it improved performance by over 70%.
Speed: Despite doing more work by mixing regions, ProPINN was actually 2 to 3 times faster than the Transformer-based models. This is because it uses a simple, efficient mixing layer instead of the heavy, complex calculations required by attention mechanisms.
Stability: In the most chaotic fluid simulations, where other models crashed or produced "NaN" (Not a Number) errors, ProPINN kept working smoothly.

The authors also showed that this method works well even as the models get bigger. Standard PINNs tend to get worse as you add more layers (making them "deeper"), but ProPINN actually gets better, showing that it scales up effectively.

Why This Matters

This paper is a big deal because it moves beyond just "trying different tricks" to solve physics problems. It identifies the fundamental reason why these AI models fail: they lack a physical connection between neighbors. By fixing the architecture to mimic how physical forces actually propagate (like the springs in a mesh), ProPINN unlocks the true potential of these neural networks.

It's a reminder that sometimes, the best way to make a smart machine smarter isn't to make it bigger or give it more data, but to change how it connects with itself. ProPINN shows that by simply letting neighbors share their load, we can solve some of the most complex equations in physics with unprecedented speed and accuracy. Whether it's designing better airplanes, predicting weather patterns, or simulating new materials, this new approach offers a reliable path forward for the future of scientific computing.

Technical Summary: ProPINN: Demystifying Propagation Failures in Physics-Informed Neural Networks

1. Problem Statement

Physics-Informed Neural Networks (PINNs) have emerged as a promising framework for solving Partial Differential Equations (PDEs) by embedding physical laws into the loss function. However, they frequently suffer from "propagation failures," a phenomenon where correct supervision from initial states or boundary conditions fails to propagate into the interior domain. Consequently, the model may converge to a trivial solution (e.g., zero) or exhibit high errors despite satisfying the equation constraints (residual loss) locally.

Previous research has identified this issue intuitively and proposed remedies such as adaptive sampling strategies (e.g., R3) or loss reweighting. However, these approaches often address symptoms rather than root causes. A formal, rigorous definition of the propagation mechanism within PINNs and a theoretical understanding of why it fails have remained underexplored. Furthermore, existing architectural improvements, such as Transformer-based models (PINNsFormer, SetPINN), often incur high computational costs without fully resolving the underlying optimization disconnects.

2. Methodology

2.1 Theoretical Analysis: From FEMs to Gradient Correlation

The authors draw a parallel between PINNs and classical Finite Element Methods (FEMs). In FEMs, the solution is updated via a stiffness matrix where adjacent nodes explicitly influence one another, ensuring active propagation. In contrast, standard PINNs treat collocation points as independent inputs processed by a shared neural network, leading to isolated optimization updates.

The paper formally defines Propagation Failure in PINNs by introducing a "stiffness" coefficient ( $D_{PINN}$ ) between two points $x$ and $x'$ . This coefficient measures the impact of a parameter update at $x$ on the output at $x'$ .

Key Theoretical Finding (Theorem 3.6): The authors prove that this stiffness coefficient is mathematically equivalent to the gradient correlation between the two points:
$D_{PINN}(x, x') = G_{u_\theta}(x, x') = \left\| \left\langle \frac{\partial u_\theta}{\partial \theta}\bigg|_x, \frac{\partial u_\theta}{\partial \theta}\bigg|_{x'} \right\rangle \right\|$
Root Cause: Propagation failure occurs when the gradient correlation between nearby collocation points is low (approaching orthogonality). This is exacerbated in high-dimensional parameter spaces where independent optimization of points leads to conflicting gradient directions.

2.2 Proposed Architecture: ProPINN

To address low gradient correlation, the authors propose ProPINN, a new architecture designed to unite region gradients without relying on computationally expensive attention mechanisms.

Differential Perturbation: Instead of processing a single point $x$ , ProPINN augments it by perturbing its position within multiscale regions. These augmented points are projected into deep representations using a shared projector $\mathcal{P}(\cdot)$ . Crucially, the gradients of these augmented points are backpropagated together, effectively aggregating gradient information from a local neighborhood.
Multi-Region Mixing: The representations from the original point and the perturbed regions are pooled and mixed via a lightweight MLP layer $\mathcal{M}(\cdot)$ . This mechanism forces the model to learn a unified representation for a region, thereby enhancing the gradient correlation among nearby points.
Efficiency: Unlike Transformer-based approaches that compute pairwise attention (quadratic complexity), ProPINN uses linear mixing and parallel computation, maintaining efficiency comparable to single-point PINNs while significantly improving performance.

3. Key Contributions

Formal Definition of Propagation Failure: The paper provides the first rigorous definition of propagation failure from a model architecture perspective, proving that it stems from low gradient correlation between adjacent points, analogous to a lack of "stiffness" in FEMs.
Gradient Correlation as a Diagnostic: The authors introduce gradient correlation as a precise, quantitative criterion to identify where and why PINNs fail, offering a more accurate diagnostic tool than residual loss maps alone.
ProPINN Architecture: A novel, efficient architecture that utilizes multi-region mixing and differential perturbation to unite region gradients. It effectively mitigates propagation failures without the computational overhead of Transformers.
Theoretical Scalability: The work demonstrates that ProPINN not only solves failure modes but also enables better scalability, whereas vanilla PINNs often degrade in performance as model size increases due to worsening gradient orthogonality.

4. Experimental Results

The authors evaluated ProPINN on six benchmarks, ranging from standard 1D/2D PDEs (Convection, 1D-Reaction, Allen-Cahn, 1D-Wave) to complex fluid dynamics tasks (Karman Vortex, Fluid Dynamics).

Standard Benchmarks: ProPINN consistently outperformed state-of-the-art models, including Vanilla PINN, QRes, FLS, KAN, PirateNet, PINNsFormer, and SetPINN.
- It achieved a 46% relative reduction in error (rMAE) compared to the second-best model across four standard tasks.
- It successfully resolved failure modes in Convection and 1D-Reaction where single-point architectures failed completely.
Complex Physics: In Navier-Stokes tasks (Karman Vortex and Fluid Dynamics), ProPINN achieved the best performance with over 30% relative improvement over previous bests. Notably, Transformer-based models (PINNsFormer, SetPINN) suffered from training instability (NaNs) in the Fluid Dynamics task, while ProPINN remained stable.
Efficiency: ProPINN was 2-3x faster than Transformer-based models and comparable in speed to single-point PINNs, while delivering a 60%+ relative error reduction.
Gradient Correlation Analysis: Visualizations confirmed that ProPINN maintains high gradient correlation throughout training, whereas Vanilla PINN's gradient correlation deteriorates, correlating with error accumulation.

5. Significance and Claims

The paper claims that ProPINN represents a fundamental shift in addressing PINN optimization challenges by moving from heuristic sampling strategies to architectural solutions grounded in the physics of propagation.

Demystification: By linking PINN failures to gradient correlation, the paper provides a theoretical foundation for diagnosing and fixing PINN failures, moving beyond intuitive descriptions.
Practicality: ProPINN offers a "favorable trade-off" between performance and efficiency, making it a viable alternative to heavy Transformer-based models for complex PDE solving.
Real-World Applicability: The successful application to Navier-Stokes equations suggests ProPINN is well-suited for real-world engineering applications involving complex fluid dynamics, such as aerodynamic simulations.

The authors conclude that while the method is highly effective for the tested PDEs, future work may be needed to address extremely high-dimensional problems (e.g., quantum physics or optimal control) where the computational cost of multi-region mixing might become prohibitive.