PhysGuard: Fisher-Guided Gradient Projection for Sim-to-Real Neural PDE Surrogates

PhysGuard is a physics-preserving framework that utilizes the empirical Fisher Information Matrix to guide gradient projection during fine-tuning, effectively reducing the sim-to-real gap in neural PDE surrogates by protecting core physical representations while adapting to experimental data.

The Gist

The Big Problem: The "Simulation vs. Reality" Gap

Imagine you teach a robot to drive a car using a perfect video game simulator. In the game, the roads are always smooth, the weather is perfect, and the physics are exact. The robot learns to drive flawlessly.

Now, you take that same robot and put it on a real road. Suddenly, there are potholes, wind gusts, and slippery leaves. The robot, trained only on the perfect game, might crash because it doesn't know how to handle the messy reality. This is called the Sim-to-Real gap.

In science, researchers use "Neural Operators" (a type of AI) to solve complex physics problems, like predicting how air flows over a wing or how fire burns. They train these AIs on massive computer simulations. But when they try to use these AIs on real-world sensor data, the results often get messy or inaccurate.

The Dilemma: To Fix or To Break?

To fix this, scientists usually try to fine-tune the AI using a small amount of real data. Think of it like giving the robot a quick driving lesson on the real road.

However, there is a catch:

• If you let the robot learn too freely, it might "unlearn" the fundamental rules of driving it learned in the simulator. It might start focusing on tiny, irrelevant details (like a specific crack in the road) and forget the big picture (like how to steer around a curve).
• In physics terms, the AI might start ignoring the large-scale, smooth patterns (like a swirling vortex of wind) and instead get confused by tiny, noisy jitters (like sensor static).

The Solution: PhysGuard

The authors propose a new method called PhysGuard. Think of it as a "Physics Bodyguard" for the AI.

Here is how it works, using a simple analogy:

1. The "Muscle Memory" Map (Fisher Information)

Imagine the AI has a huge library of "muscle memory" stored in its brain. Some of these memories are critical for understanding the core laws of physics (like gravity or fluid flow), while others are just minor details.

PhysGuard uses a mathematical tool called the Fisher Information Matrix to scan the AI's brain before it starts learning from real data. It asks: "Which parts of your brain are absolutely essential for keeping the physics correct?"

It turns out that for these physics AIs, the most important knowledge is concentrated in just a few specific directions, like the main highways in a city. The rest of the brain is like side streets where you can drive freely without causing traffic jams.

2. The "Guardrail" (Gradient Projection)

When the AI starts learning from the real data, PhysGuard acts like a smart guardrail system.

• If the AI tries to make a change that would damage those critical "main highway" memories (the core physics), PhysGuard blocks that change.
• If the AI wants to make a change to the "side streets" (the parts that don't affect the core physics), PhysGuard lets it go.

This ensures the AI can adapt to the messy real world without forgetting the fundamental rules it learned in the simulator.

3. The "Smart Filter" (Low-Frequency vs. High-Frequency)

The paper makes a fascinating discovery: The "critical memories" that PhysGuard protects are specifically the ones that handle low-frequency information.

• Low-frequency = Big, smooth, slow-moving patterns (like a giant ocean wave).
• High-frequency = Tiny, fast, jittery details (like the ripples on the surface of the wave).

PhysGuard realizes that the AI needs to keep the "big wave" patterns intact because that's the real physics. It allows the AI to adjust the "ripples" to match the real-world noise, but it strictly protects the "wave" structure.

Why This Matters (The Results)

The researchers tested PhysGuard on four different types of AI architectures and three different physical scenarios (like wind flowing around a cylinder and turbulent fire).

• The Result: PhysGuard consistently performed better than standard methods.
• The Analogy: When the gap between the simulator and reality was huge (a very bumpy road), standard methods often made the AI worse at understanding the big picture. PhysGuard, however, kept the AI's understanding of the big picture intact while still allowing it to learn the new road conditions.
• The Stat: In the worst-case scenarios, PhysGuard reduced errors in the "big picture" physics by up to 32% compared to standard fine-tuning.

Summary

PhysGuard is a safety net for AI scientists. It lets AI models learn from real-world data without accidentally deleting the most important physics lessons they learned in the simulation. It does this by identifying the "core knowledge" directions in the AI's brain and gently guiding the learning process so it never overwrites them.

The paper notes that this works best for models where the "core knowledge" is concentrated in a few specific areas (which was true for the models they tested). If a model's knowledge is spread out too evenly, this "guardrail" method might not be as effective.

Technical Summary

Technical Summary: PhysGuard

1. Problem Statement

Neural operator models, which learn mappings between function spaces to serve as surrogates for Partial Differential Equations (PDEs), have shown promise in scientific machine learning. However, a significant "sim-to-real gap" exists: models trained on idealized simulation data often degrade in performance when applied to real-world experimental measurements. This degradation stems from discrepancies such as sensor noise, unmodeled physical effects, and systematic domain shifts.

Standard fine-tuning on limited real data is a direct solution but carries a critical risk: unconstrained optimization can overwrite the core physics-relevant representations learned during pretraining. Specifically, the model may prioritize fitting high-frequency noise in the real data while neglecting large-scale, coherent physical structures (e.g., vortex streets or mean flow profiles) that encode the system's governing physics. While knowledge-preserving adaptation techniques (e.g., EWC, GPM, LoRA) exist for vision and language tasks, their applicability to neural operators is unclear due to fundamental architectural differences. Neural operators rely on mechanisms like spectral convolutions and branch-trunk factorizations to preserve PDE physics, rather than semantic or visual features.

2. Methodology: PhysGuard

The authors propose PhysGuard, a framework for sim-to-real adaptation that preserves pretrained physical knowledge by restricting fine-tuning updates to directions that do not interfere with physics-critical parameters.

Core Mechanism: Fisher-Guided Gradient Projection

PhysGuard operates on the insight that physical knowledge in a pretrained neural operator is concentrated along a small number of principal directions in the parameter space. The method proceeds in two phases:

1. Physics Subspace Identification (Offline):

   • The method computes the empirical Fisher Information Matrix (FIM) using a subset of the simulation dataset. The FIM identifies parameter directions where the loss is most sensitive; these correspond to the directions encoding the core low-frequency physical structures.
   • To handle models with millions of parameters, PhysGuard employs a Gram-matrix formulation. Instead of decomposing the massive $d \times d$ FIM (where $d$ is the parameter count), it computes the SVD of the much smaller $N \times N$ Gram matrix ($K = GG^\top$), where $N$ is the number of samples. The eigenvectors of the FIM in parameter space are recovered from the eigenvectors of the Gram matrix.
   • An adaptive threshold selects the top-$k$ eigenvectors that capture a cumulative fraction (e.g., 90%) of the total Fisher information, defining the "physics-critical subspace."

2. Constrained Fine-Tuning (Online):

   • During fine-tuning on real data, gradient updates are projected onto the orthogonal complement (null space) of the identified physics-critical subspace.
   • For a gradient $g$ and subspace basis $U$, the projected gradient is $g_{proj} = g - \alpha U U^\top g$, where $\alpha \in [0, 1]$ controls the protection strength.
   • This ensures that updates adapt the model to real data without overwriting the low-frequency physical structures encoded during pretraining. The process is layer-wise, with each layer maintaining its own subspace basis.

Handling Complex Parameters

For architectures like Fourier Neural Operators (FNO) that utilize complex-valued spectral weights, PhysGuard decomposes complex gradients into real and imaginary components to perform the projection within standard real-valued linear algebra, ensuring consistency with Wirtinger calculus.

3. Key Contributions

• PhysGuard Framework: A novel, architecture-agnostic framework for sim-to-real adaptation of neural operators that preserves pretrained physics without requiring auxiliary loss terms or sensitive hyperparameters.
• Efficient Subspace Estimation: The introduction of a Gram-matrix kernel formulation that makes subspace estimation tractable for models with millions of parameters, avoiding the computational infeasibility of direct FIM eigendecomposition.
• Spectral Insight: Through a "spectral probe" experiment, the authors demonstrate that the dominant eigenvectors of the FIM for neural operators are strongly associated with low-frequency output structures. This confirms that protecting these directions preserves the large-scale coherent patterns essential to PDE physics.
• Systematic Evaluation: The first systematic study of knowledge-preserving fine-tuning for neural PDE surrogates, evaluated across four distinct architectures (FNO, CNO, DeepONet, Transolver) and three physical scenarios on the RealPDEBench benchmark.

4. Experimental Results

Experiments were conducted on RealPDEBench, which pairs numerical simulations with real experimental data for cylinder flow, controlled cylinder flow, and turbulent combustion.

• Performance: PhysGuard ranked first in 38 out of 48 metric-architecture-scenario combinations. It consistently outperformed baselines including Direct Fine-Tuning (DFT), L2-SP, and EWC.
• Low-Frequency Preservation: The most significant gains were observed in preserving low-frequency physical structures. Compared to standard fine-tuning, PhysGuard reduced low-frequency error by 22% to 32% across architectures. In contrast, DFT sometimes degraded low-frequency performance (e.g., increasing error for CNO on cylinder flow) despite improving overall RMSE.
• Domain Shift Sensitivity: The benefits of PhysGuard scaled with the severity of the domain shift. Under severe shifts (e.g., Cylinder Flow), it provided substantial improvements in $R^2$ (e.g., >50% relative improvement for DeepONet), whereas baselines often failed or degraded performance.
• Subspace Efficiency: The protected subspace was found to be compact, typically comprising only 1.0% to 12.0% of the total parameter directions, leaving the majority of the parameter space free for adaptation.

Note on Limitations: The authors note that PhysGuard relies on the assumption of a low-rank Fisher spectrum. On the foundation-scale operator DPOT-S, where the Fisher spectrum is nearly uniform (lacking a clear low-rank structure), PhysGuard offered limited benefits, as the protected subspace effectively consumed most update directions.

5. Significance and Claims

The paper claims that PhysGuard offers a principled and efficient approach to bridging the sim-to-real gap for neural PDE surrogates. By explicitly identifying and protecting the parameter directions responsible for low-frequency physical structures, the method enables models to adapt to real-world noise and variability without "forgetting" the core physics learned during pretraining.

The authors position this work as a step toward more reliable and resource-efficient deployment of scientific machine learning models. By reducing the need to train surrogates from scratch for every new experimental setup, PhysGuard aims to lower the data and compute costs associated with applying neural operators to fields such as fluid dynamics, climate emulation, and materials modeling. The framework is presented as a transparent, interpretable alternative to black-box fine-tuning, providing practitioners with a geometric handle on which aspects of the pretrained physics are being preserved.