🔢 mathematics

Smoothness Errors in Dynamics Models and How to Avoid Them

This paper identifies that strict unitary graph convolutions hinder dynamics modeling by overconstraining natural smoothing processes, and proposes a new method of relaxed unitary convolutions generalized to meshes that outperforms existing baselines in solving partial differential equations and weather forecasting.

Original authors: Edward Berman, Luisa Li, Jung Yeon Park, Robin Walters

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Edward Berman, Luisa Li, Jung Yeon Park, Robin Walters

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Predicting the Future of Fluids and Heat

Imagine you are trying to predict how a drop of ink spreads in a glass of water, how heat travels through a metal armadillo shell, or how weather patterns move across the Earth. These are all Partial Differential Equations (PDEs). For decades, scientists have used complex math to solve these, but it's slow.

Recently, researchers started using Neural Networks (AI) to do this faster. They turn the physical object (like the Earth or a metal part) into a digital mesh—a web of connected dots (nodes). The AI learns how the dots talk to each other to predict the future.

However, the paper argues that the current AI models have a specific flaw: they get the "smoothness" wrong.

The Problem: The "Too Smooth" vs. "Too Rough" Dilemma

To understand the problem, imagine you are trying to copy a painting of a sunset.

The "Oversmoothing" Problem (The Blender):
Standard AI models (Graph Neural Networks) act like a blender. Every time they look at the painting, they mix the colors of neighboring dots together. After a few rounds, the distinct edges of the sunset clouds disappear, and the whole picture becomes a blurry, uniform orange blob. The AI has smoothed out the details it was supposed to keep.
The "Undersmoothing" Problem (The Rigid Sculptor):
To fix the blender issue, a previous team of researchers (Kiani et al., 2024) invented a new type of AI called Unitary Convolutions. Think of this as a rigid sculptor who is forbidden from ever changing the total "amount of paint" or the "roughness" of the surface. They promise to keep the painting exactly as sharp as it started.
- The Catch: In the real world, some things are supposed to get smoother. Heat should spread out and blur. Ink should diffuse. By forcing the AI to keep the image perfectly sharp (preserving "smoothness" mathematically), this rigid sculptor fails to model heat or diffusion correctly. It's like trying to model a melting ice cube by insisting the ice stays perfectly hard.

The Solution: The "Relaxed" Sculptor

The authors of this paper propose a middle ground. They realized that for physical systems like heat flow, we need a model that can be rigid when necessary but flexible when the physics demands it.

They created a new model called R-UNIMESH (Relaxed Unitary Mesh).

How it works (The Analogy):
Imagine you are baking a cake (the physical system).

The Old Rigid Model: It insists the batter stays in a perfect, unmoving cube shape forever. It can't bake.
The Old Blurry Model: It lets the batter spill everywhere until it's a flat puddle.
The New R-UNIMESH: It starts with a rigid framework (the Unitary part) to keep the structure stable, but it has a special "release valve" (the Relaxation).
- When the physics says "keep it sharp" (like in a wave), the model stays rigid.
- When the physics says "let it spread" (like heat), the model opens the valve and allows the smoothness to increase naturally.

They achieved this "valve" using two main tricks:

Taylor Truncation: Instead of doing a perfect, infinite calculation to keep things rigid, they stop the math early (truncating the series). This introduces just enough "wiggle room" to let the heat flow.
Compositional Breaking: They build the model in layers. Some layers are the "rigid sculptors" that preserve structure, and they are followed by "breaker" layers that are allowed to change the smoothness to match reality.

Why "Mesh" Matters

Most AI models work on simple grids (like a chessboard). But real objects (like the Earth, a human heart, or a car engine) are irregular 3D shapes. The authors extended their math to work on Meshes (triangular webs that wrap around 3D objects).

They had to invent a new way to measure "smoothness" on these 3D shapes (using something called the Rayleigh Quotient adapted for meshes) to ensure their model knew exactly how much smoothing was happening at every step.

What They Found (The Results)

The team tested their new "Relaxed" model against the old "Blender" models and the "Rigid Sculptor" models on three types of challenges:

Heat Diffusion: Predicting how heat spreads on a complex metal shape.
- Result: The Rigid model failed because it wouldn't let the heat spread. The Blender model smoothed it too much. R-UNIMESH won because it let the heat spread exactly as much as physics required.
Wave Propagation: Predicting how waves move.
- Result: R-UNIMESH performed just as well as the best existing models, proving it doesn't break when things don't need to smooth out.
Weather Forecasting: Predicting temperature and pressure on a global map.
- Result: Even though they used a smaller computer budget than the giants in the field, their model was competitive with state-of-the-art weather predictors, showing that getting the "smoothness" right helps even in massive real-world tasks.

The Takeaway

The paper argues that in the world of AI physics, one size does not fit all.

If you force an AI to be perfectly rigid (Unitary), it fails at diffusion (heat/ink).
If you let it be too loose, it loses all detail.
The Sweet Spot: You need a model that knows when to be rigid and when to let things smooth out. By building this "Relaxed Unitary" flexibility into the AI, they created a tool that predicts the future of fluids, heat, and weather more accurately than previous methods.

In short: They fixed the AI's "smoothness error" by teaching it that sometimes, things should get blurry, and that's okay.

Technical Summary: Smoothness Errors in Dynamics Models and How to Avoid Them

Problem Statement

Modern neural networks, particularly Graph Neural Networks (GNNs), have shown promise for solving Partial Differential Equations (PDEs) on surfaces by discretizing domains into meshes. However, standard GNNs suffer from oversmoothing, where node features become increasingly similar to their neighbors over successive message-passing iterations, degrading the model's ability to capture complex dynamics.

To counteract this, recent work (Kiani et al., 2024) proposed unitary graph convolutions, which mathematically constrain weight matrices to be unitary ( $U^\dagger U = I$ ). This ensures the preservation of the Rayleigh quotient, a measure of signal smoothness on graphs, thereby preventing oversmoothing. However, this paper identifies a critical limitation: many physical systems (e.g., heat diffusion) naturally require increased smoothness over time. In such cases, unitary constraints are overconstraining, leading to undersmoothing where the model fails to replicate the natural dissipation or smoothing inherent in the physical process. The core problem is the lack of a mechanism to balance the preservation of structural information with the necessary natural smoothing required by specific dynamical systems.

Methodology

The authors propose a systematic approach to characterize and resolve these smoothness errors through theoretical analysis and architectural relaxation.

1. Theoretical Characterization

The paper derives a lower bound on the approximation error of unitary functions. Using group invariance theory, the authors prove that unitary functions are overly restrictive when the norm of the ground-truth function has high angular dependence. Specifically, unitary transformations preserve the norm of data points on concentric spheres in the feature space; if the target dynamics require changes in this norm (e.g., energy dissipation in heat diffusion), unitary models cannot approximate the solution accurately.

2. Relaxed Unitary Convolutions

To address the overconstraining nature of strict unitarity, the authors introduce Relaxed Unitary Convolutions. Two specific relaxation strategies are proposed:

Taylor Truncation: Instead of computing the full matrix exponential required for Lie unitary convolutions, the authors truncate the Taylor series expansion at a finite order $T_{max}$ . This allows for precise control over the degree of relaxation. A small $T_{max}$ introduces non-unitary behavior, enabling the model to alter the Rayleigh quotient and capture natural smoothing.
Compositional Smoothness-Breaking: To scale unitary models without increasing depth (which causes training instability), the authors propose a compositional architecture. This involves:
1. Zero-padding input features to increase dimensionality (which trivially preserves norms and the Rayleigh quotient).
2. Passing the signal through a smoothness-preserving component ( $P$ ) consisting of $k$ layers of unitary mesh convolutions.
3. Passing the signal through a smoothness-breaking component ( $B$ ), such as an MLP or GCN, which maps to the target dimension and breaks the unitary constraint, allowing the model to learn the necessary smoothing dynamics.

3. Generalization to Meshes

The framework is extended from graphs to meshes (2D manifolds embedded in $\mathbb{R}^3$ ).

Mesh Rayleigh Quotient: The authors generalize the Rayleigh quotient using the Robust Laplacian (Sharp & Crane, 2020), which employs cotangent weights and edge rewiring to satisfy the intrinsic Delaunay criterion, ensuring valid smoothness measurement even on non-manifold or irregular meshes.
Unitary Mesh Convolution: Unitary convolutions are adapted to use the weighted adjacency matrix derived from the Robust Laplacian. The authors prove that under the assumption that mesh weights obey the Delaunay criterion, these convolutions preserve the Mesh Rayleigh Quotient.

The resulting models are named R-UNIGRAPH (for graph-based tasks using Taylor truncation) and R-UNIMESH (for mesh-based tasks using the compositional approach).

Key Contributions

Theoretical Bounds: Derivation of a lower bound on the approximation error of unitary functions, demonstrating that they are overconstrained for dynamical systems where the solution's norm exhibits high angular dependence (Section 4).
Architectural Innovation: Introduction of Relaxed Unitary Convolutions that balance accuracy with smoothness preservation. This includes extending the Rayleigh quotient and unitary convolution framework from graphs to meshes (Section 5).
Empirical Analysis: Systematic investigation of smoothness tendencies in various mesh-GNN architectures. The study shows that controlling smoothness is a key factor in successful modeling, providing equal or greater improvements compared to other inductive biases like equivariance (Section 6).

Experimental Results

The authors evaluate their methods on three primary benchmarks:

Heat Diffusion on Grid Graphs:
- Task: Predicting heat distribution over time.
- Result: R-UNIGRAPH significantly outperforms both standard GCNs (which oversmooth) and strict Lie unitary models (which undersmooth). It achieves the lowest Mean Rayleigh Quotient Error (MRE), demonstrating its ability to match the true smoothness of the target signal (Table 1, Fig. 3).
Dynamics on Complex Meshes (MeshPDE):
- Tasks: Solving Heat, Wave, and Cahn-Hilliard equations on intricate PyVista meshes.
- Baselines: Compared against Mesh-aware Transformers, Equivariant Neural Networks (GemCNN, EMAN, Hermes, EGNN), and standard GNNs.
- Result: R-UNIMESH achieves state-of-the-art performance on diffusive dynamics (Heat equation), outperforming all baselines in NRMSE, SMAPE, and Rayleigh Error (RE). It captures the true smoothness of the solution almost exactly at every timestep (Table 2, Fig. 4). While performance on Wave and Cahn-Hilliard is competitive, the gains are most pronounced for diffusive systems. The results suggest that geometric inductive biases (like unitarity) are crucial for generalization to unseen meshes with complex geometries.
Weather Forecasting (WeatherBench2):
- Task: Autoregressive prediction of temperature (T850) and geopotential (Z500) on an Earth mesh.
- Result: R-UNIMESH remains competitive with state-of-the-art models in RMSE and Anomaly Correlation Coefficient (ACC), particularly at early lead times. However, the authors note that in this setting (where cross-mesh generalization is not the primary test), the advantages of geometric inductive biases are less significant compared to the MeshPDE tasks (Fig. 5, Table 13).

Significance and Claims

The paper claims that smoothness errors (both oversmoothing and undersmoothing) are a fundamental limitation in current GNN-based dynamics modeling. By theoretically characterizing the limitations of unitary models and proposing relaxed unitary convolutions, the authors provide a mechanism to adapt the model's smoothness behavior to the specific physics of the problem.

The significance lies in demonstrating that:

Strictly preserving smoothness (via unitarity) is detrimental for systems requiring natural smoothing (e.g., diffusion).
Approximately unitary networks can successfully model these dynamics by balancing norm preservation with the flexibility to alter the Rayleigh quotient.
This approach yields superior performance on diffusive PDEs compared to strong baselines, including equivariant networks and transformers, without requiring massive computational resources.

The authors modestly note limitations: the method is most effective for diffusive systems (Heat equation) and less so for highly oscillatory (Wave) or discontinuous (Cahn-Hilliard) systems, where global smoothness metrics may not fully capture the dynamics. They also acknowledge that their weather forecasting results are a proof-of-concept for studying inductive biases rather than a claim of state-of-the-art operational forecasting performance. Future work is suggested for incorporating boundary conditions and using dynamic truncation orders ( $T_{max}$ ) for oscillatory systems.