🤖 machine learning

Do Neural Operators Forget Geometry? The Forgetting Hypothesis in Deep Operator Learning

This paper introduces the "Geometric Forgetting Hypothesis," demonstrating that deep neural operators progressively lose domain geometry due to their Markovian structure, which degrades performance, and proposes a lightweight memory injection mechanism to restore geometric fidelity and improve stability.

Original authors: Yanming Xia, Angelica I. Aviles-Rivero

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yanming Xia, Angelica I. Aviles-Rivero

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Amnesia" of AI Physics Engines

Imagine you are teaching a robot to predict how water flows around a rock in a river. To do this, you give the robot a map of the river and the rock (the geometry) and ask it to simulate the water.

In the world of "Neural Operators" (AI models designed to solve physics problems), researchers noticed something strange. These models work great on simple, square-shaped grids, but they often fail when the shapes get weird or irregular. They seem to "forget" what the rock or the riverbank looks like as they process the information.

The authors of this paper propose a new theory called the Geometric Forgetting Hypothesis. They argue that this isn't just a bug in how the data is entered; it's a fundamental flaw in how these deep AI brains are built.

The Core Problem: The "Telephone Game" of Physics

Think of a standard Neural Operator like a game of Telephone (or "Broken Telephone").

You whisper a secret (the shape of the river) to the first person (Layer 1).
They whisper it to the next person (Layer 2), who whispers it to Layer 3, and so on, until the last person (the Output).

In these AI models, each layer mixes the information globally (like everyone in the room shouting at once to blend their voices). The paper argues that with every step down the line, the specific details of the "shape" get muddied and diluted. By the time the message reaches the end, the AI has statistically "forgotten" the original shape of the domain. It knows the physics of water, but it has lost the map of where the water is supposed to go.

The Two Types of "Forgetting"

The paper found that different types of AI models forget in different ways:

The "Blurry Photo" (Fourier-based models like FNO): These models are good at seeing the big picture but lose the sharp edges. Imagine taking a photo of a jagged rock and applying a "blur" filter over and over again. Eventually, the sharp edges of the rock disappear, and the AI can't tell where the rock ends and the water begins.
The "Ghost Rock" (Attention-based models like Transolver): These models are even worse at irregular shapes. They might look at the water flow but completely ignore the obstacle. It's as if the AI sees the water flowing, but the rock has become invisible to it.

The Solution: The "Memory Injection"

To fix this, the authors introduced a simple trick called Geometry Memory Injection.

Instead of just telling the AI the shape of the river at the very beginning (Input), they "remind" the AI of the shape at every single step of the process.

Analogy: Imagine you are walking through a dark forest with a guide. Instead of just telling you the map at the start, the guide stops at every turn and says, "Remember, the cliff is to your left."
The Result: By constantly re-injecting the geometric information, the AI doesn't lose the map. The paper shows this makes the models much more accurate and stable, especially on weird shapes.

The "Shortcut" Trap

One of the most interesting discoveries in the paper is a phenomenon they call the Geometric Shortcut.

If you only remind the AI of the shape at the very end of the process (instead of throughout), the AI gets lazy.

The Analogy: Imagine a student taking a math test. If the teacher whispers the answer key only in the last 10 seconds, the student stops trying to solve the math problems and just writes down the answer key.
The Result: The AI stops learning the actual physics (the hard work) and just relies on the shape data at the very end to guess the answer. This leads to a "feature collapse" where the model fails miserably because it didn't actually learn the rules of the game. The paper found that for some models, you must inject the shape information early and often to prevent this cheating.

The "Laplace" Exception

The authors also tested a specific type of model called the Laplace Neural Operator (LNO).

The Finding: This model didn't need the "Memory Injection" at all.
Why? The authors explain that the LNO is built differently. It has "intrinsic memory" built into its math (like a sponge that naturally holds water). Because it naturally remembers the shape and boundaries, adding extra reminders didn't help it much. This proves that the "forgetting" problem is specific to the other models, not a general law of physics.

Summary

The paper claims that deep AI models for physics are structurally prone to losing track of shapes as they get deeper. This isn't a design choice; it's a mathematical inevitability of how they process information. To fix it, you can't just rely on a good starting map; you have to constantly remind the AI of the shape throughout its entire thinking process. If you don't, the AI will either blur the edges or ignore the obstacles entirely.

Technical Summary: The Forgetting Hypothesis in Deep Operator Learning

Problem Statement
Neural operators have emerged as a powerful paradigm for learning solution operators of Partial Differential Equations (PDEs), offering zero-shot evaluation capabilities. However, their ability to generalize to irregular geometries remains a significant challenge. While existing approaches focus on sophisticated input encoding strategies (e.g., coordinate transformations, graph-based methods, or tokenizing geometry), the behavior of these models as information propagates through deep layers is poorly understood. The authors identify a critical gap: it is unclear whether standard neural operator architectures inherently retain geometric information or if they progressively lose it as depth increases.

Methodology
The paper formalizes the Geometric Forgetting Hypothesis, positing that the Markovian structure of standard neural operator layers, combined with reliance on global mixing mechanisms (such as Fast Fourier Transforms in FNOs or self-attention in Transformers), leads to a progressive loss of geometric information.

Theoretical Framework: The authors model the neural operator as a Markov chain where the hidden state $V_{l+1}$ depends only on the previous state $V_l$ and not explicitly on the domain geometry $G$ . By applying the Data Processing Inequality, they prove that mutual information $I(G; V_l)$ is non-increasing with depth. Consequently, without explicit reintroduction of geometric constraints, deep representations approach statistical independence from the domain geometry.
Diagnostic Probes: To quantify this forgetting, the authors introduce two measurable proxies:
- Layer-wise Geometric Fidelity: An auxiliary decoder reconstructs the domain mask from latent features at each layer. An increasing Mean Squared Error (MSE) in reconstruction indicates macroscopic geometric information loss.
- Spectral Power Distribution: This analyzes the frequency content of latent features to detect the loss of boundary information (high-frequency components corresponding to sharp boundaries).
Geometry Memory Injection: To counteract forgetting, the authors propose a lightweight mechanism that breaks the Markov chain by explicitly injecting geometry $G$ into intermediate layers via an update rule $V_{l+1} = M_l(O_l(V_l), G)$ . They evaluate three integration strategies: Affine Modulation (FiLM), Additive Injection, and Concatenation.
Control Case: The study utilizes the Laplace Neural Operator (LNO) as a control. Unlike Fourier-based methods, LNO utilizes a pole-residue formulation that mathematically encodes boundary conditions, theoretically providing "intrinsic memory" of geometry.

Key Results
Experiments were conducted on diverse fluid dynamics benchmarks (FlowBench, AirfRANS, Darcy) involving irregular domains and zero-shot generalization tasks.

Validation of Forgetting: Both spectral (FNO) and attention-based (Transolver) operators demonstrated systematic geometric information decay.
- FNO: Suffered from the loss of high-frequency boundary information, leading to smoothed boundaries in predictions.
- Transolver: Exhibited "macroscopic geometric collapse," failing to reconstruct the overall domain shape in deeper layers.
Efficacy of Memory Injection: Introducing geometry memory injection consistently reduced relative $L_2$ errors across all datasets. For instance, on the LDC-NSHT dataset, FNO error dropped from $4.11 \times 10^{-1}$ to $2.39 \times 10^{-1}$ , and Transolver from $2.73 \times 10^{-1}$ to $2.30 \times 10^{-1}$ .
Intrinsic vs. Extrinsic Memory: The LNO control case showed minimal performance gains from external memory injection, confirming that its mathematical formulation inherently preserves geometric information. This validates that the forgetting observed in FNO and Transolver is a structural deficiency, not a general capacity limitation.
The Geometric Shortcut: A critical instability was discovered in Transformer-based operators (Transolver). When geometry is injected only at the final layer, the optimizer exploits this explicit information to bypass the physics backbone entirely. Gradient analysis revealed that the final layer saturates with gradient signals while upstream layers collapse, leading to feature collapse and catastrophic error increases. This phenomenon was not observed in FNOs, which remained robust to injection depth.

Significance and Claims
The paper claims to provide the first principled explanation for why continuous geometric conditioning is necessary in deep operator learning. The authors argue that geometry retention is a structural requirement rather than a mere design choice.

Structural Limitation: The study reveals that standard architectures are predisposed to "wash out" geometric information due to their global mixing operations.
Architectural Necessity: The findings suggest that relying solely on input encoding is insufficient; geometric constraints must be actively maintained throughout the network depth to prevent information dissipation.
Stability Insight: The discovery of the "Geometric Shortcut" in attention-based models highlights a fundamental divergence in how spectral and attention-based architectures handle information flow, necessitating early integration of geometry for stability in Transformer-based operators.

The authors conclude that future research should shift focus from complex input encoders and heuristic injection strategies toward principled architectural mechanisms that actively preserve geometric memory throughout the network.