🤖 machine learning

Integral regularization PINNs for evolution equations

This paper proposes Integral Regularization PINNs (IR-PINNs), a novel method that enhances long-time integration accuracy for evolution equations by incorporating integral-based residual constraints over sub-intervals and utilizing adaptive sampling to dynamically refine collocation points.

Original authors: Xiaodong Feng, Haojiong Shangguan, Tao Tang, Xiaoliang Wan

Published 2026-04-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiaodong Feng, Haojiong Shangguan, Tao Tang, Xiaoliang Wan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the weather for the next month. You have a super-smart computer model (a neural network) that knows the laws of physics. However, there's a catch: as the model predicts day by day, tiny mistakes start to pile up. By day 10, the forecast is slightly off; by day 20, it's completely wrong. It's like trying to walk a tightrope while blindfolded; one small stumble early on sends you tumbling off the edge.

This is the problem scientists face with Evolution Equations (mathematical formulas that describe how things change over time, like heat spreading, fluids flowing, or populations growing). Standard AI methods, called PINNs (Physics-Informed Neural Networks), are great at learning these laws, but they struggle with "long-time integration." They tend to forget the past, leading to a cascade of errors.

This paper introduces a new solution called IR-PINNs (Integral Regularization PINNs). Here is how it works, using simple analogies:

1. The Problem: The "Point-by-Point" Blind Spot

Imagine you are trying to learn a song by listening to individual notes played one by one, with no connection between them. You might get the pitch of each note right, but you won't understand the melody or the rhythm.

Standard PINNs work like this. They look at specific moments in time (collocation points) and try to make sure the physics holds true at that exact instant. They treat time as a series of isolated snapshots. Because they don't explicitly check how one moment flows into the next, small errors in the beginning get magnified as time goes on, causing the whole prediction to collapse.

2. The Solution: The "Group Photo" Approach (Integral Regularization)

The authors propose a new trick: Integral Regularization.

Instead of just checking the physics at single points, IR-PINNs look at chunks of time (sub-intervals) and ask: "Does the total change over this whole chunk of time make sense?"

The Analogy: Imagine you are driving a car from New York to Boston.
- Old Method (PINNs): You check your speedometer every 10 seconds. If you are going 60 mph at 10:00:00 and 60 mph at 10:00:10, you think you are doing fine. But you might have missed a huge detour in between.
- New Method (IR-PINNs): You check the odometer at the start of the hour and the end of the hour. You calculate the total distance covered. If the math says you should have traveled 60 miles, but your odometer says 100, you know something is wrong, even if your speedometer looked perfect at every single second.

By forcing the AI to satisfy the "total distance" (the integral) over small time chunks, the model is forced to respect the connection between the past and the future. It prevents the errors from drifting apart.

3. The Bonus Feature: Adaptive Sampling (The "Flashlight")

Even with the new method, some parts of the solution are harder to predict than others. Maybe a fluid is swirling violently in one spot, while it's calm elsewhere.

The paper adds an Adaptive Sampling strategy.

The Analogy: Imagine you are looking for a lost key in a dark room with a flashlight. A standard approach is to shine the light in a grid pattern, evenly spaced.
The IR-PINN approach: The AI acts like a smart flashlight. It realizes, "Hey, the key is likely to be in this messy pile of clothes where the light is dim." So, it moves the flashlight to shine brighter and more frequently on the chaotic, high-stress areas (sharp gradients) and shines less on the empty, calm floor.

This ensures the AI spends its brainpower exactly where it's needed most, refining the solution in the most difficult regions.

Why Does This Matter?

The authors tested this on some very difficult problems:

The Lorenz System: A classic "chaos" model (like the butterfly effect) where tiny changes lead to massive differences.
Fluid Dynamics: Simulating how flames or waves move chaotically.
Probability: Predicting how a cloud of particles spreads out over time.

The Result: In every test, the new IR-PINNs were significantly more accurate over long periods than the old methods. They didn't just "guess" the future; they respected the continuous flow of time, keeping the prediction stable and reliable.

In a Nutshell

The paper says: "Don't just look at the present moment. Look at the journey between moments. By checking the 'total story' of a time interval rather than just individual sentences, and by shining a brighter light on the confusing parts, we can make AI models that can predict the future for a long time without losing their way."

1. Problem Statement

Evolution equations, encompassing both Ordinary Differential Equations (ODEs) and Partial Differential Equations (PDEs), are fundamental for modeling dynamic systems in physics, biology, and engineering. While Physics-Informed Neural Networks (PINNs) offer a mesh-free framework for solving these equations, they face a critical limitation: temporal error accumulation during long-time integration.

The Core Issue: Standard PINNs treat temporal collocation points in isolation. They minimize the residual of the differential equation at specific points but fail to explicitly enforce the strong temporal correlations inherent in evolution equations (where the state at time $t$ depends on the state at $t-\Delta t$ ).
Consequence: This leads to overfitting on training points and a failure to capture long-time dynamics, particularly in stiff systems or chaotic regimes (e.g., Lorentz system, Kuramoto-Sivashinsky equation). The error often grows exponentially over time, rendering the solution inaccurate for long-duration simulations.

2. Methodology: Integral Regularization PINNs (IR-PINNs)

The authors propose Integral Regularization PINNs (IR-PINNs), a novel framework that enhances standard PINNs by incorporating an integral-based residual term into the loss function to enforce temporal consistency.

A. Integral Formulation

Instead of relying solely on the differential form $u_t + \mathcal{N}[u] = f$ , the method reformulates the equation over a temporal sub-interval $[a, b]$ :
$u(b,x) - u(a,x) + \int_a^b (\mathcal{N}[u](t,x) - f(t,x)) \, dt = 0$
This integral form inherently links the solution states at the boundaries of the interval ( $a$ and $b$ ) with the behavior of the system throughout the interval.

B. Loss Function Construction

The total loss function is augmented with an integral regularization term:
$\mathcal{L}_{total} = \mathcal{L}_{r} + \alpha \mathcal{L}_{int} + \mathcal{L}_{ic} + \mathcal{L}_{bc}$
Where:

$\mathcal{L}_{r}$ : Standard differential residual loss.
$\mathcal{L}_{int}$ $L_{in t}$ : The new integral-based residual loss. The time domain $[0, T]$ $[0, T]$ is divided into $N$ $N$ sub-intervals. Two variants are proposed:
1. IR-PINNs1: Enforces the integral constraint over each local sub-interval $[t_k, t_{k+1}]$ .
2. IR-PINNs2: Enforces the integral constraint from the initial time $t_0$ to the current time $t_{k+1}$ (accumulating history).
Hyperparameter $\alpha$ : Set to 1 based on the Mean Value Theorem, which suggests the differential and integral residuals are of similar magnitude.
Computational Efficiency: The method reuses precomputed values of the differential operator to evaluate the integral term, ensuring negligible additional computational cost.

C. Adaptive Sampling Strategy

To further handle complex scenarios (sharp gradients, unbounded domains), the authors integrate an adaptive sampling strategy:

A continuous joint probability density model $p(t,x)$ is constructed to approximate the distribution of the squared residual $r^2(t,x)$ .
The model uses a bounded polynomial layer for time and a bounded KR-net for space.
New collocation points are generated via importance sampling from regions with high residuals, dynamically refining the training set to focus on areas of rapid variation.

3. Key Contributions

Novel Framework (IR-PINNs): Introduction of an integral regularization term that explicitly strengthens temporal correlations, addressing the root cause of error accumulation in standard PINNs.
Two Variants: Proposal of two specific formulations (local vs. cumulative integral constraints) to suit different problem characteristics.
Adaptive Integration: Extension of IR-PINNs with an adaptive sampling strategy that dynamically refines collocation points based on evolving solution residuals.
Comprehensive Benchmarking: Rigorous validation on four distinct benchmark problems covering ODEs, PDEs, chaotic systems, and unbounded domains.

4. Numerical Results

The paper evaluates IR-PINNs against standard PINNs and Causal PINNs on four benchmarks:

Benchmark	System Type	Key Findings
Lorentz System	Chaotic ODE	IR-PINNs reduced relative $L_2$ error by two orders of magnitude compared to standard PINNs (e.g., $3.5 \times 10^{-3}$ vs $5.5 \times 10^{-1}$ ). IR-PINNs successfully captured the chaotic trajectory over $T=20$ , whereas standard PINNs diverged.
Kuramoto-Sivashinsky	Chaotic PDE	Standard PINNs failed due to error explosion in the $[0.1, 0.2]$ time window. IR-PINNs controlled error growth, achieving errors of $\approx 4 \times 10^{-3}$ compared to $1.1 \times 10^{-1}$ for standard PINNs. Outperformed Causal PINNs in both accuracy and training efficiency.
Boussinesq-Burgers	Coupled PDE	Demonstrated the efficacy of adaptive sampling. While the integral regularization provided modest gains for smooth solutions, the combination with adaptive sampling significantly improved accuracy in regions with sharp gradients.
Fokker-Planck Equation	Unbounded Domain	Applied to a 2D probability density evolution problem using Temporal Normalizing Flows. IR-PINNs combined with adaptive sampling successfully maintained probability normalization and non-negativity, achieving lower KL divergence than standard PINNs.

General Observation: In all cases, IR-PINNs (both variants) demonstrated superior long-time stability and accuracy. The computational cost increased only slightly (approx. 10-20% in running time) due to the integral term, which is negligible compared to the massive gain in accuracy.

5. Significance

Solving Long-Time Integration: This work provides a robust solution to the "long-time integration" bottleneck in PINNs, a major hurdle for applying deep learning to dynamic systems.
Bridging Numerical Analysis and Deep Learning: By incorporating the integral form (a concept from classical numerical analysis) into the neural network loss, the method leverages the "memory" of the system, mimicking the stability properties of traditional time-stepping schemes while retaining the mesh-free flexibility of PINNs.
Scalability: The method is applicable to a wide range of problems, from stiff ODEs to high-dimensional PDEs and stochastic processes, offering a unified framework for evolution equations.
Future Impact: The approach opens new avenues for solving complex dynamic systems in fluid dynamics, material science, and statistical physics where long-term prediction accuracy is critical.