🔢 mathematics

Multi-Grade Deep Learning for Partial Differential Equations with Applications to the Burgers Equation

This paper proposes a two-stage multi-grade deep learning (TS-MGDL) method that progressively trains shallow networks to capture low-to-high frequency components and then refines them hierarchically, effectively overcoming optimization challenges in nonlinear PDEs and significantly reducing predictive errors for the viscous Burgers' equation compared to single-grade learning.

Original authors: Yuesheng Xu, Taishan Zeng

Published 2026-06-16

📖 4 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuesheng Xu, Taishan Zeng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a student to draw a very complex, jagged mountain range.

The Problem: The "All-at-Once" Struggle
Traditionally, if you wanted a neural network (a type of computer brain) to solve a difficult math problem like the Burgers equation (which models things like fluid flow and shock waves), you would build a single, massive, deep network and try to train it all at once.

The paper compares this to handing the student a blank canvas and saying, "Draw the whole mountain, the clouds, the rocks, and the tiny cracks in the stone, all in one go."

The Issue: The student gets overwhelmed. They get good at drawing the big, smooth shapes (the low-frequency parts), but they struggle terribly with the sharp, jagged details (the high-frequency parts). They might get stuck in a "local minimum," which is like the student thinking, "I've drawn a decent hill, so I'll just stop here," even though the real mountain has a sharp peak they missed. This is called spectral bias.

The Solution: The "Two-Stage Multi-Grade" Method
The authors propose a new way to teach the computer, called TS-MGDL (Two-Stage Multi-Grade Deep Learning). They break the massive task into smaller, manageable steps, like a school curriculum.

Stage 1: The "Grade-by-Grade" Approach

Instead of one giant class, imagine a school with three grades (Grade 1, Grade 2, Grade 3).

Grade 1 (The Basics): You start with a small, simple network. It learns the broad, smooth outline of the mountain. Once it does a decent job, you freeze its brain. You don't let it change anymore. It has learned the "low-frequency" parts.
Grade 2 (The Details): You take the frozen Grade 1 network and attach a new, slightly bigger network on top of it. This new network doesn't try to redraw the whole mountain. It only looks at what Grade 1 missed (the "residual" or the error). It learns the medium-sized bumps and rocks. Once it's done, you freeze Grade 2 as well.
Grade 3 (The Fine Print): You attach a third network. This one focuses only on the tiny, sharp cracks and the very steep cliffs that the first two grades missed.

The Analogy: Think of it like sculpting.

Grade 1 carves the rough block of stone into a general human shape.
Grade 2 refines the muscles and limbs.
Grade 3 carves the facial features and the texture of the skin.
By freezing the previous grades, you ensure the new student doesn't accidentally mess up the work the previous students did.

Stage 2: The "Final Polish" (Fine-Tuning)

After the three grades are done, the authors realized there was still room for improvement. The "frozen" parts were good, but maybe they could be tweaked slightly to fit the new layers better.

So, they unlocked the last few layers of the entire chain (the most recent grades) and let them retrain together.

The Analogy: Imagine the sculptor looking at the finished statue. They say, "The face is great, and the muscles are good, but if I tweak the shoulder slightly, the whole pose will look more natural." They make small, coordinated adjustments to the final layers to smooth out any remaining rough edges.

Why This Works (The Results)

The paper tested this method on the Burgers equation in 1D, 2D, and 3D. This equation is tricky because it involves "shocks"—sudden, violent changes in the data (like a sonic boom or a traffic jam).

The Old Way (Single-Grade Learning): The computer struggled to find the sharp shocks. It either missed them entirely or produced a wobbly, inaccurate solution.
The New Way (TS-MGDL): By building the solution layer by layer, the computer handled the sharp shocks much better.
The Score: The new method reduced the error by a factor of up to 60 compared to the old method. In simple terms, the new method was 60 times more accurate.

Key Takeaways

Don't try to learn everything at once. Break complex problems into a sequence of simpler steps (Grades).
Freeze what works. Once a part of the solution is learned well, lock it in so the next part doesn't ruin it.
Focus on the mistakes. Each new step only learns what the previous steps got wrong (the residual).
Do a final check. Unlock the most recent parts and polish them together for the best possible result.

The paper concludes that this "Two-Stage Multi-Grade" approach is a powerful new tool for solving difficult, nonlinear math problems that involve sharp changes, outperforming traditional deep learning methods significantly.

Technical Summary: Two-Stage Multi-Grade Deep Learning for Nonlinear PDEs

Problem Statement
Deep neural networks (DNNs) have emerged as powerful tools for solving partial differential equations (PDEs), particularly through Physics-Informed Neural Networks (PINNs). However, training deep architectures to solve nonlinear PDEs presents significant challenges. As network depth increases, the associated optimization landscape becomes highly non-convex and complex, often leading to difficulties in convergence. A primary obstacle is "spectral bias" (or the frequency principle), where DNNs naturally learn low-frequency components of a function more rapidly than high-frequency components or steep gradients. This limitation is particularly detrimental when solving nonlinear PDEs, such as the viscous Burgers equation, which are characterized by sharp gradients, shock waves, and multiscale features. Traditional single-grade learning (SGL), which trains all parameters of a deep network simultaneously, often struggles to capture these fine details accurately and efficiently.

Methodology: Two-Stage Multi-Grade Deep Learning (TS-MGDL)
To address these optimization and approximation challenges, the authors propose a Two-Stage Multi-Grade Deep Learning (TS-MGDL) framework. This method decomposes the learning process into a hierarchical, sequential structure rather than a monolithic end-to-end optimization.

Multi-Grade Deep Learning (MGDL) - Stage 1:
The core of the method involves training the network "grade by grade."
- Grade 1: A shallow neural network is trained to approximate the target solution.
- Subsequent Grades: For each new grade $\ell+1$ , a new shallow network is trained to approximate the residual (the error) of the cumulative solution formed by the previous grades. Crucially, the parameters of the networks from previous grades are frozen and treated as fixed "adaptive basis" functions. The new grade only optimizes its own parameters to minimize the remaining loss.
- Mechanism: This approach transforms the problem of learning a complex function into a sequence of simpler residual learning tasks. By freezing previous layers, the method mitigates gradient interference and reduces the complexity of the optimization problem at each step.
Two-Stage Refinement - Stage 2:
Recognizing that the greedy, sequential freezing of parameters in Stage 1 might limit the network's ability to find the global optimum, a second stage is introduced.
- Unfreezing: After the multi-grade training is complete, a subset of the layers (specifically the last $k$ layers of the final grade and potentially some layers from preceding grades) are "unfrozen."
- Retraining: These selected layers are retrained jointly using the Stage 1 solution as an initialization. This step allows for a broader search of the parameter space to refine the solution and correct any suboptimal local minima established during the greedy training phase.

Theoretical Contributions
The paper provides theoretical guarantees for the proposed method:

Monotonic Loss Reduction: The authors prove that under appropriate optimization strategies, the loss function decreases monotonically with each added grade in Stage 1 (Proposition 1) and further decreases in Stage 2 (Proposition 3).
Strict Decrease: Under conditions of differentiability and non-zero gradients, the loss is shown to strictly decrease as the algorithm progresses from one grade to the next (Proposition 2) and from Stage 1 to Stage 2 (Proposition 4).
Comparison to SGL: The framework is contrasted with ResNets, Relay Backpropagation, Curriculum Learning, and Gradient Boosting, highlighting that TS-MGDL offers a unique hierarchical, compositional approach that decomposes the problem structurally rather than just data-wise or via simultaneous multi-loss optimization.

Numerical Results
The efficacy of TS-MGDL is validated through numerical experiments on the 1D, 2D, and 3D viscous Burgers equations, which are benchmark problems known for their nonlinear propagation and shock formation.

Performance: TS-MGDL significantly outperforms Single-Grade Learning (SGL) baselines. In the 1D case, TS-MGDL reduced the relative $L^2$ error by a factor of approximately 60 compared to the best SGL configuration (reaching $9.65 \times 10^{-6}$ vs. $2.53 \times 10^{-4}$ ). Similar improvements were observed in 2D and 3D scenarios.
Convergence: The training loss curves for TS-MGDL exhibit smooth, rapid descent without the significant oscillations often seen in SGL methods, indicating more stable convergence.
Robustness: Experiments across multiple random seeds and hybrid optimization strategies (combining Adam with L-BFGS) confirmed that TS-MGDL maintains superior accuracy and stability compared to deep SGL networks, even when the latter are augmented with second-order optimization.
Loss Weighting: A notable finding is that TS-MGDL performs effectively with unweighted loss functions ( $\lambda_I = \lambda_B = 1$ ), whereas standard PINNs often require careful tuning of penalty weights to balance PDE, initial, and boundary condition losses.

Significance and Claims
The paper claims that TS-MGDL offers a robust solution to the spectral bias and optimization difficulties inherent in deep learning for PDEs. By structurally decomposing the learning task into a sequence of shallow sub-problems and refining the result through a second stage, the method achieves:

Accelerated Convergence: Mitigating vanishing/exploding gradients and stabilizing the optimization process.
Implicit Regularization: Preventing overfitting by gradually increasing model complexity and utilizing early stopping-like mechanisms in the first stage.
Intrinsic Multiscale Representation: Naturally capturing both low-frequency global features (in early grades) and high-frequency local details (in later grades), which is critical for problems with sharp gradients.
Superior Accuracy: Achieving significantly lower predictive errors than traditional deep networks, even those with comparable depth or enhanced with regularization and advanced optimizers.

The authors conclude that while the method is demonstrated on the Burgers equation, its architecture holds promise for broader applications to other nonlinear PDEs, effectively addressing the limitations of traditional single-grade deep learning models.

Stage 1: The "Grade-by-Grade" Approach

Stage 2: The "Final Polish" (Fine-Tuning)

Why This Works (The Results)

Key Takeaways

More like this