💻 computer science

Watching Physics: the Generative Science of Matter and Motion

This paper proposes the "Generative Sciences of Matter and Motion," a new framework that integrates visual data, experiments, and high-fidelity simulations to transform generative video models into scientifically valid instruments for inferring, predicting, and designing the physics of matter in motion.

Original authors: Hagen Holthusen, Kevin Linka, Ellen Kuhl

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hagen Holthusen, Kevin Linka, Ellen Kuhl

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magic camera that can watch the world and then instantly create new movies of things happening. For a long time, scientists have been trying to figure out: If this camera watches a rubber ball squish or a heart beat, can it actually understand the physics behind it, or is it just a really good actor mimicking the look?

This paper argues that the answer is: It depends on what you're watching, and we need to teach the camera some real science to make it trustworthy.

Here is the breakdown of their idea, using simple analogies:

The Big Idea: Three New "Sciences"

The authors propose a new way of doing science called the "Generative Sciences of Matter and Motion." They break this down into three parts, like a three-step recipe for a perfect meal:

Simulogenics (The Chef's Recipe): This is the old way. We write down strict mathematical rules (like Newton's laws) to simulate how things move. It's like following a recipe exactly. It's accurate, but it's slow and requires a lot of math.
Physiogenics (The Food Critic): This is the new AI way. We show the AI thousands of videos of things moving, and it learns to guess what happens next just by watching. It's like a critic who has eaten at every restaurant and can guess the menu just by smelling the air. It's fast and looks great, but it might be guessing wrong about the ingredients.
Materiogenics (The Master Chef): This is the goal. We combine the Chef's strict rules with the Critic's speed. We teach the AI the laws of physics so it doesn't just look real; it is real.

The Three Tests: From Easy to Impossible

To prove their point, the team tested this "magic camera" (using a powerful AI video generator called Sora) on three different scenarios, getting harder each time.

1. The Rubber Block (The Easy Test)

The Scene: Squeezing a block of rubber. It stretches and bulges out the sides.
The Result: The AI did a great job! Because the rubber moves smoothly and you can see exactly how it stretches, the AI learned the pattern perfectly.
The Analogy: It's like watching a dancer spin. If the dancer's movements are smooth and visible, a camera can easily copy the dance. The AI could even measure how much the rubber stretched just by looking at the video.
Verdict: Success. When physics is visible, the AI works.

2. The Soda Can (The Tricky Test)

The Scene: Crushing an aluminum soda can. It crumples, folds, and buckles in complex ways.
The Result: The AI made a video that looked like a crushed can, but it was physically wrong. It didn't know why the can folded where it did.
The Analogy: Imagine watching a magician make a can disappear. The AI can copy the trick (the can vanishes), but it doesn't know the mechanism (the secret compartment). If you asked the AI to crush the can with a different amount of force, it would probably guess wrong because it doesn't understand the metal's internal strength or the "crunch" inside.
Verdict: Failure. When the physics happens inside the object (hidden stress and folding), the AI just fakes the look. It's a "plausible lie."

3. The Human Heart (The Impossible Test)

The Scene: A heart beating, pumping blood, and contracting muscles.
The Result: The AI made a video of a beating heart that looked okay, but the geometry was wrong, and the internal forces were nonsense.
The Analogy: This is like trying to predict the weather just by looking at a painting of a storm. You see the rain and clouds, but you don't know the temperature, pressure, or wind speed. The heart is driven by invisible electrical signals and muscle fibers. The AI can mimic the rhythm, but it can't simulate the biology.
Verdict: Dangerous. Without strict physics rules, the AI creates "hallucinations" that look real but could lead to bad medical decisions.

The Solution: The "Physics-Teacher"

The paper concludes that we can't just let AI watch videos and guess. We need to give it a Physics Teacher.

The Hybrid Approach: We need to feed the AI not just videos, but also computer simulations (the strict math) and real experiments.
The Goal: We want an AI that can generate a video of a heart beating, but one that also knows the exact pressure inside the chambers and the stress on the muscle fibers.

Why Does This Matter?

If we can build this "Physics-AI," it changes everything:

Virtual Labs: Instead of building a physical prototype of a new car or a new material, we could ask the AI to generate 1,000 versions of it crashing, and it would tell us exactly which one is strongest.
Personalized Medicine: We could take a blurry MRI of a patient's heart, feed it to the AI, and have it generate a perfect, physics-accurate digital twin to test surgeries before they happen.
Design: We could say, "Design a material that is as light as a feather but as strong as steel," and the AI would generate the structure that makes that possible.

In short: The paper says, "AI is great at faking the look of reality, but to be useful for science and engineering, we need to teach it the rules of reality." We are moving from "Visual Realism" (it looks good) to "Physical Validity" (it actually works).

1. Problem Statement

The paper addresses a critical gap in the application of Artificial Intelligence to physics and engineering: Can generative models trained on visual data (images/video) infer the underlying physical laws governing matter and motion, or do they merely reproduce visual realism without physical validity?

The Limitation of Current AI: Modern generative models (e.g., video diffusion models) excel at creating visually plausible content by learning latent structures from observational data. However, they often lack physical admissibility. They may generate motion that looks correct but violates conservation laws, constitutive relationships, or internal state variables (e.g., stress, plastic strain).
The Data Disconnect: While large-scale video datasets contain rich dynamics, they rarely encode explicit constitutive responses, state variables, loading paths, or boundary conditions. Conversely, traditional physics-based simulations (like Finite Element Methods) provide precise internal fields but are computationally expensive and require explicit equations.
The Core Question: How can we bridge the gap between unstructured visual observation and rigorous physical simulation to create models that are both visually realistic and physically consistent?

2. Methodology

The authors propose a new framework called the Generative Sciences of Matter and Motion, which unifies three sub-disciplines:

Simulogenics: Learning to generate physically consistent trajectories from data (inverting the traditional simulation process).
Physiogenics: Inferring physical laws directly from visual observations of motion.
Materiogenics: Designing matter with prescribed properties by navigating high-dimensional design spaces using generative models.

Experimental Framework:
To test the limits of visual learning, the authors employed a staged benchmark using OpenAI's Sora (a generative video model) to generate visual data. They compared three modalities across three scenarios of increasing complexity:

Generative AI Video: Visual baseline capturing macroscopic kinematics.
Digital Image Correlation (DIC): Applied to the generated video to extract measurable surface strain fields (using the tool PYVALE).
High-Fidelity Finite Element Method (FEM): Served as the mechanical reference, solving governing equations (balance of momentum, constitutive laws) to provide full-field internal data (stress, strain energy, reaction forces).

Three Test Cases:

Rubber Compression (Uniaxial): A smooth, hyperelastic deformation where physics is largely visible on the surface.
Can Crushing (Thin-walled Collapse): A nonlinear instability problem involving plasticity, buckling, and contact, where internal state variables dominate.
Cardiac Motion (Beating Heart): A high-dimensional biomechanical system involving active contraction, anisotropy, and fluid-structure interaction.

3. Key Results

The study identifies distinct regimes where visual learning succeeds, fails, or requires supervision:

Regime 1: Visible Kinematics (Rubber Compression)
- Result: In the early stages of deformation, the generative model successfully captured the macroscopic "hourglass" shape. DIC applied to the generated video yielded strain fields that closely matched the FEM reference.
- Limitation: As deformation increased, temporal coherence in the video degraded, causing feature tracking to fail. The model captured the shape but not the precise particle trajectories.
- Conclusion: When physics manifests clearly in surface kinematics, generative models can serve as "virtual experiments" for calibration and data augmentation.
Regime 2: Internal State Dominance (Can Crushing)
- Result: The generative model reproduced the visual narrative of buckling and folding (morphology) but failed to ensure mechanical admissibility.
- Failure Mode: The model interpolated learned deformation patterns without enforcing equilibrium or constitutive constraints. It could not guarantee correct responses under modified loading or material parameters because the governing physics (plasticity, instability) are not directly observable.
- Conclusion: Visual plausibility is insufficient when internal state variables (stress, plastic strain) drive the behavior.
Regime 3: Complex Biomechanics (Cardiac Motion)
- Result: The generative model captured the global kinematic cycle (filling, contraction, relaxation) but failed to reproduce accurate anatomical geometry or enforce mechanical consistency. The strain fields were illustrative rather than physically derived.
- Conclusion: For complex living systems, active stress generation and fiber architecture are invisible to the camera. Pure visual learning cannot capture the constitutive structure.

4. Key Contributions

Conceptual Framework: The paper defines the Generative Sciences of Matter and Motion, unifying Simulogenics, Physiogenics, and Materiogenics into a single paradigm that bridges observation, simulation, and generation.
Empirical Benchmarking: It provides the first systematic evaluation of generative video models against physical ground truth (FEM and experiments) across varying levels of complexity.
Identification of Regimes: It delineates specific boundaries where visual learning is sufficient (smooth kinematics) versus where it fails (instability, plasticity, active mechanics), establishing that visual realism $\neq$ physical validity.
Proposed Hybrid Architecture: The authors advocate for physics-grounded foundation models. These models should use high-fidelity simulations and experiments as "structured supervision" to guide generative models, ensuring that generated outputs satisfy physical laws (e.g., conservation of momentum, constitutive admissibility).

5. Significance and Outlook

From Prediction to Design: The framework shifts the role of AI from merely predicting motion to designing matter. By inverting learned relationships, these models could enable the generation of material configurations that achieve targeted physical behaviors.
Scientific Instrument: Physics-consistent generative models can transform into instruments for inference, prediction, and design. They allow for scalable, patient-specific digital twins (e.g., in cardiology) where sparse clinical data conditions a physics-informed model to reconstruct high-resolution, mechanically consistent simulations.
Bridging the Gap: The work suggests a future where "learning simulation without equations" is possible, provided the generative process is constrained by mechanistic supervision. This reduces the experimental burden while preserving mechanical interpretability, enabling a new era of virtual experimentation and inverse material characterization.

In summary, the paper argues that for AI to truly understand the physics of matter and motion, it must move beyond appearance to mechanism, integrating visual data with the rigorous constraints of physics-based simulation.