🔬 physics

Mosaic: A Benchmark Suite for Differentiable Physics Solvers

The paper introduces Mosaic, an open-source benchmarking framework that standardizes the evaluation of differentiable PDE solvers by packaging them as containerized components, revealing significant variations in computational cost, numerical conditioning, and compatibility across 14 solvers while demonstrating that all gradient-producing solvers converge to similar optima.

Original authors: Andrin Rehmann, Heiko Zimmermann, Dion Häfner

Published 2026-06-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Andrin Rehmann, Heiko Zimmermann, Dion Häfner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to invent a new recipe. You have a kitchen full of different ovens (the physics solvers). Some are high-tech smart ovens, some are old-school gas ranges, and some are fancy convection units. They all bake bread, but they do it in very different ways.

Now, imagine you want to use a robot arm to tweak the recipe automatically. To do this, the robot needs to know exactly how changing the temperature or the flour amount changes the final taste. In the world of physics simulations, this "taste feedback" is called a gradient.

For a long time, scientists have had these ovens, but no one knew which one would give the robot the most accurate feedback, how much electricity it would cost to get that feedback, or if the oven would even explode if you asked it the wrong question.

Enter "Mosaic."

Think of Mosaic as a universal testing kitchen. The researchers built a standardized setup where they can plug in 14 different "ovens" (computer programs that solve physics equations) and test them side-by-side. They wrapped each oven in a special box (called a Tesseract) so that no matter what language the oven was built in (Python, Julia, C++) or how it works inside, the robot arm can ask it the exact same question in the exact same way.

Here is what they discovered when they started baking:

1. The "Feedback" Varies Wildly

Just like some ovens take 5 minutes to preheat and others take 30, the cost of getting that "taste feedback" (the gradient) varies massively.

The Speedsters: Some modern ovens (like those built on JAX or PyTorch) can give you the feedback almost instantly, with almost no extra effort. It's like having a smart oven that tells you the temperature the second you open the door.
The Heavy Lifters: Other ovens (like OpenFOAM or deal.II) are powerful but require a massive amount of manual work to get that feedback. It's like having to manually measure the heat with a thermometer for every single degree change. The researchers found that for some of these, writing the code to get the feedback took hundreds or even thousands of extra lines of code.

2. The "Stability" Trap

Some ovens are great for baking a simple loaf of bread (a smooth, easy physics problem), but if you try to bake a complex soufflé (a chaotic fluid flow problem), they might collapse.

The paper found that for some tasks, like optimizing the shape of a cylinder to reduce drag in water, half the ovens simply couldn't be used at all. They were structurally incompatible. It's like trying to use a toaster to bake a cake; the machine just isn't built for that shape of problem.
Even among the ones that could work, some were "unstable." If you asked them to simulate a long time period, the numbers would get so messy (a mathematical concept called ill-conditioning) that the robot arm would start spinning in circles, unable to find the best solution.

3. The "Perfect" Oven Doesn't Exist

The biggest surprise? Accuracy wasn't the main problem.
Most of the ovens, once they actually managed to give feedback, were surprisingly accurate. If they didn't crash, they usually found the same "perfect recipe" (the optimal solution).

The real bottlenecks weren't that the math was wrong; it was that the memory ran out (the oven got too hot), the setup was too hard to configure, or the cost was too high.
It's like having 10 chefs who can all cook a perfect steak. The problem isn't that they can't cook; it's that three of them need a $10,000 kitchen, one of them needs a 24-hour prep time, and one of them refuses to cook unless you give them a specific type of knife.

4. The "Negative Control" (The Simple Test)

To make sure they were testing the ovens and not the ingredients, they started with a very simple task: heating a flat metal plate. This is the "negative control."

On this simple task, every single oven that could run it produced the same perfect result. This proved that the differences they saw in the complex tasks weren't because the math was broken; it was because the complex tasks exposed the specific weaknesses of each oven's design.

The Bottom Line

The paper introduces Mosaic not as a new way to solve physics problems, but as a standardized menu for researchers.

Before Mosaic, picking a solver was like guessing which oven to buy based on a salesperson's word.
With Mosaic, you can look at the menu and see: "If I need to do this specific task, Oven A is 200 times faster than Oven B, but Oven B handles my specific shape better."

The researchers conclude that for many problems, the "best" solver isn't the one with the most accurate math, but the one that fits your specific constraints (like memory limits or setup time) without crashing. They made all their "kitchen tools" and "menus" open-source so anyone can add their own ovens to the test kitchen and see how they stack up.

Technical Summary: Mosaic: A Benchmark Suite for Differentiable Physics Solvers

Problem Statement

While differentiable partial differential equation (PDE) solvers are increasingly advertised as enabling technologies for solver-in-the-loop machine learning, gradient-based optimal control, and inverse problems, the practical costs of obtaining correct and usable gradients remain opaque. The landscape of differentiable solvers is fragmented across languages, automatic differentiation (AD) frameworks, and numerical methods. Practitioners face significant uncertainty regarding:

Integration Effort: The varying expertise required to integrate solvers ranging from turnkey applications to low-level kernel toolkits.
Gradient Quality: The lack of systematic measurement regarding gradient accuracy, numerical conditioning, and stability.
Structural Compatibility: Solver-specific constraints (e.g., periodic-only boundary conditions, incompatible penalization schemes) that often only surface after significant commitment to a specific stack.

Current benchmarks either focus on forward accuracy of neural surrogates (e.g., APEBench, PDEBench) or compare AD backends within a single framework (e.g., PhiFlow), failing to systematically measure gradient properties across independent solver implementations.

Methodology

The authors introduce Mosaic, an extensible benchmarking framework designed to standardize access to solver gradients and enable cross-solver comparison. The methodology rests on three core artifacts:

Containerized Solver Interface (Tesseract):
- Solvers are packaged as Docker containers (Tessera) exposing a uniform API regardless of language or AD strategy.
- The interface defines two entry points: apply(inputs) → outputs (forward pass) and vector_jacobian_product(inputs, cotangents) → vjp (reverse-mode gradient).
- This abstraction allows solvers to be swapped via configuration strings without rewriting evaluation code. Forward-only solvers (e.g., OpenFOAM, deal.II) are automatically wrapped to provide finite-difference gradients as a reference baseline.
Benchmark Tasks:
- Four physical domains are evaluated, ordered by difficulty:
  - Heat Transfer (H): Conductivity inversion (Negative control; smooth physics isolates AD bugs).
  - Structural Mechanics (S): Compliance minimization via SIMP topology optimization.
  - 2D Fluids (F2): Drag minimization via inflow optimization around a cylinder (Non-periodic, obstacle-laden).
  - 3D Fluids (F3): Initial condition recovery in a periodic Taylor-Green vortex box.
- Tasks are designed as standalone optimization problems without learned components, ensuring that failures can be attributed to the solver's gradient mechanism rather than model approximation errors.
Evaluation Protocol:
- Gradient Accuracy: Measured via cosine similarity and relative $\ell_2$ error against central finite differences at optimal perturbation sizes.
- Conditioning: Analyzed via the singular value spectrum of the Jacobian.
- Performance: Wall-clock time for forward and VJP passes, memory consumption, and the VJP/forward cost ratio.
- Optimization Convergence: Fixed iteration budgets with Adam and L-BFGS optimizers to measure convergence to the global optimum.

The suite includes 14 solver backends spanning diverse AD strategies (source transformation, tape-based adjoints, hand-derived adjoints) and numerical methods (finite difference, finite volume, finite elements, lattice Boltzmann, spectral).

Key Results

The evaluation of 14 solvers across the four domains reveals several critical findings:

Gradient Accuracy vs. Conditioning: All differentiable solvers that produce gradients achieve high accuracy (cosine similarity > 0.999) against finite differences. However, Jacobian conditioning varies by orders of magnitude. Projection-based solvers exhibit steep spectral decay (poor conditioning) compared to lattice Boltzmann or spectral methods, which impacts the stability of second-order optimizers.
Computational Cost: There is an order-of-magnitude variation in computational cost. Source-transformation AD (e.g., JAX-based solvers) maintains a nearly constant VJP/forward ratio, while tape-based adjoints (e.g., FEniCS, Firedrake) scale more steeply with problem size.
Structural Incompatibility: On the most complex task (2D drag minimization with obstacles), only 3 of 7 fluid backends could represent the problem due to boundary condition or penalization incompatibilities. This "attrition" occurs before gradient quality becomes a factor.
Optimization Outcomes: Despite vast differences in conditioning and cost, all solvers that successfully run converge to similar optima on well-conditioned problems (Heat, Structural). On the 3D IC recovery task, L-BFGS with solenoidal projection significantly outperforms Adam and unprojected L-BFGS, reducing initial condition error from >40% to <0.5%.
Implementation Effort: Integration complexity varies wildly. Solvers built on modern AD frameworks (JAX, PyTorch) often require zero additional lines of code for gradients, whereas low-level kernel toolkits (e.g., NVIDIA Warp) require hundreds of bespoke lines for adjoint implementation.

Significance and Claims

The paper positions Mosaic not as a source of new empirical results for a single solver, but as infrastructure that lowers the barrier to entry for differentiable physics research, analogous to OpenAI Gym for reinforcement learning.

Standardization: Mosaic provides the first standardized interface to access, compare, and trust gradients across independent solver implementations.
Practitioner Guidance: The benchmark surfaces practical barriers (memory limits, numerical stability, setup compatibility) that are more critical than raw gradient accuracy alone. It suggests that for well-conditioned problems, the fastest solver with the lowest integration effort is the optimal choice.
Community Extensibility: The framework is open-source and designed to evolve. It allows researchers to stress-test failure modes before committing to a specific solver stack and enables the construction of solver-in-the-loop pipelines on top of a uniform interface.
Validation of AD: The results validate that while hand-derived adjoints can be faster, reverse-mode AD through modern frameworks offers the best effort-to-capability tradeoff, providing sufficient accuracy for optimization at a fraction of the implementation cost.

The authors conclude that while gradient quality is generally sufficient, the choice of solver backend is a decisive factor in whether gradient-based workflows succeed or fail, primarily due to structural compatibility and numerical conditioning rather than the correctness of the gradient itself.