💻 computer science

PhysGM: Large Physical Gaussian Model for Feed-Forward 4D Synthesis

PhysGM is a feed-forward framework that jointly predicts 3D Gaussian representations and physical properties from a single image to enable immediate, high-fidelity 4D simulation, leveraging a new dataset called PhysAssets and Direct Preference Optimization to overcome the limitations of slow, optimization-heavy, and physically disconnected prior methods.

Original authors: Chunji Lv, Zequn Chen, Donglin Di, Weinan Zhang, Hao Li, Wei Chen, Yinjie Lei, Changsheng Li

Published 2026-03-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chunji Lv, Zequn Chen, Donglin Di, Weinan Zhang, Hao Li, Wei Chen, Yinjie Lei, Changsheng Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a single photograph of a toy, a piece of fruit, or a lump of clay. Right now, if you want to see what happens when you drop that object or squish it, you usually have to hire a team of engineers. They spend hours building a 3D model, manually telling the computer "this part is hard like steel" and "this part is soft like jelly," and then running a slow, complex physics simulation to see how it moves.

PhysGM is like a magic "instant movie maker" that skips all that hard work. It looks at your single photo and, in less than a second, figures out not just what the object looks like in 3D, but also what it feels like physically. Then, it instantly simulates a realistic video of that object falling, bouncing, or squishing.

Here is how it works, broken down with some everyday analogies:

1. The Problem: The "Slow Cooker" vs. The "Microwave"

Old Way (The Slow Cooker): Previous methods were like slow-cooking a stew. You had to gather many photos of the object from different angles, spend hours "optimizing" (tweaking) the 3D model, and then manually tell the computer the physics rules. It was slow, expensive, and couldn't be done on the fly.
PhysGM (The Microwave): This new method is a microwave. You put in one photo, press a button, and poof—you get a full 3D simulation with physics in under a minute. It doesn't "cook" the answer step-by-step; it just knows the answer because it has "studied" enough examples to recognize the pattern instantly.

2. The Secret Sauce: "Gaussians" (The Cloud of Dots)

To make the 3D object, PhysGM uses something called 3D Gaussian Splatting.

The Analogy: Imagine the object isn't made of solid plastic, but is actually a cloud of millions of tiny, glowing, fuzzy balls (like a cloud of cotton candy or a swarm of fireflies).
Each "ball" has a position, a color, and a size.
PhysGM predicts exactly where to place these fuzzy balls to recreate the shape of your object. But here's the trick: it also predicts how "squishy" or "stiff" each ball is.

3. The Brain: Learning from a Giant Library (PhysAssets)

How does the computer know that a photo of a rock means "hard and heavy" while a photo of a marshmallow means "soft and bouncy"?

The Library: The researchers built a massive library called PhysAssets. It contains over 50,000 3D objects.
The Annotation: For every single object in this library, they didn't just save the picture; they saved the "physics recipe." They labeled them with real-world data: "This is metal, it's very stiff," or "This is jelly, it's very squishy."
The Training: The AI studied this library. It learned that when it sees a shiny, gray texture, it should predict "High Stiffness." When it sees a dull, pink texture, it should predict "Low Stiffness."

4. The Two-Step Training: "School" and "Practice"

The model learns in two stages, similar to how a student learns to drive:

Stage 1 (Classroom): The AI is taught the rules. It looks at a photo and guesses the 3D shape and the physics properties. It gets graded on how well its guess matches the real data. This gives it a solid foundation.
Stage 2 (Driving School / DPO): This is the clever part. The AI generates a few different versions of a falling video. A "judge" (a computer program) compares them to a perfect reference video and says, "Version A fell too fast, Version B bounced too high, but Version C was perfect."
- The AI learns from this feedback without needing to re-calculate everything from scratch. It just learns to prefer the "winning" guesses. This is called Direct Preference Optimization (DPO).

5. The Result: Instant Physics

Once the AI is trained, here is the magic workflow:

Input: You upload one photo of a jelly donut.
Prediction: In under a second, the AI says, "Okay, that's a donut shape, and it's made of jelly material (soft, high compressibility)."
Simulation: It instantly runs a physics engine (called MPM) that treats the object like a real jelly donut.
Output: You get a video of the donut falling, hitting the floor, and wobbling realistically, all generated in about a minute.

Why This Matters

Speed: It turns a process that used to take hours into a process that takes seconds.
Realism: It doesn't just make the object move; it makes it move correctly based on its material. A steel ball bounces; a clay ball splats. PhysGM knows the difference.
Future Use: Imagine video games where you can upload a photo of a real-world object and instantly play with it in a physics world, or robots that can learn how to handle new objects just by looking at a picture.

In short: PhysGM is like a psychic chef who can look at a single photo of an ingredient and instantly know exactly how to cook it, how it will taste, and how it will react to heat, serving you a perfect meal in seconds.

1. Problem Statement

Current methods for generating physics-based 4D content (dynamic 3D scenes) face three critical bottlenecks:

Inefficiency: Most approaches rely on slow, per-scene iterative optimization. They typically require dense multi-view images to reconstruct a 3D Gaussian Splatting (3DGS) model, followed by manual specification of physical properties, and finally running a physics simulation. This process is computationally prohibitive for real-time or large-scale applications.
Suboptimal Physics Integration: Existing methods often naively concatenate pre-built 3DGS models with physics modules, ignoring physical cues embedded in the visual appearance. Alternatively, they use Score Distillation Sampling (SDS) from video models to guide physics, which is unstable, expensive, and requires differentiable physics simulators.
Lack of Data: There is a scarcity of large-scale datasets that pair 3D assets with annotated physical properties and corresponding reference simulation videos, hindering the training of generalizable generative models.

The core question addressed is: Can we bypass per-scene optimization entirely and learn a generative model that produces a complete, physically-grounded 4D simulation from a single image in a single feed-forward pass?

2. Methodology: PhysGM Framework

PhysGM is a feed-forward, transformer-based framework that jointly predicts 3D Gaussian parameters and physical properties from a single input image. The pipeline consists of four main components:

A. Model Architecture

Input Processing: The model accepts a single RGB image. To provide geometric context, it uses an MVAdapter to synthesize three auxiliary views (rear, left, right), creating a 4-view input set.
Multi-Modality Tokenization:
- Image Encoding: Uses DINOv3 to extract image features.
- Camera Encoding: Represents camera rays using Plücker coordinates processed by a dense encoder.
- Global Tokens: Three learnable global tokens are prepended to the sequence to aggregate scene-wide information specifically for physics prediction.
Transformer Backbone: A 24-layer transformer processes the concatenated tokens to learn contextualized representations.
Dual Prediction Heads:
1. DPT Head (3DGS): A Dense Prediction Transformer head that regresses 3D Gaussian parameters (position, scale, rotation, opacity, spherical harmonics) for geometry and appearance.
2. Physics Head: Predicts a distribution over physical properties: Material Class ( $C$ ), Young's Modulus ( $E$ ), and Poisson's Ratio ( $\nu$ ). It outputs mean and log-variance to model uncertainty, allowing for probabilistic sampling.

B. Physics Simulation (MPM)

The predicted parameters drive a Material Point Method (MPM) simulator:

Coupling: There is a one-to-one correspondence between MPM material points and 3D Gaussian primitives.
Deformation: The MPM simulation updates particle positions and deformation gradients ( $F_p$ ). These are mapped back to the 3D Gaussians: the particle position defines the Gaussian mean, and the polar decomposition of the deformation gradient defines the Gaussian's rotation and anisotropic scaling.
Constitutive Models: The predicted material class selects the appropriate physics model (e.g., Neo-Hookean for jelly, Fixed Corotational for metal, Drucker-Prager for sand/snow).

C. Two-Stage Training Paradigm

To achieve high fidelity without per-scene optimization, PhysGM employs a novel training strategy:

Stage 1: Supervised Pre-training: The model is trained on the PhysAssets dataset to jointly minimize reconstruction loss (MSE, LPIPS) and physical property prediction errors. This establishes a robust "physical prior."
Stage 2: Direct Preference Optimization (DPO):
- Instead of using differentiable SDS, the model is fine-tuned using preference pairs.
- For a given scene, the model samples multiple sets of physical parameters.
- Each set is simulated and rendered. The resulting videos are compared against a Ground Truth (GT) reference video using a perceptual metric (trajectory alignment via SAM-2 and CoTracker-3).
- The "winner" (closest to GT) and "loser" (furthest) pairs are used to optimize the model via DPO loss, aligning the generative distribution with physically plausible outcomes without requiring a differentiable simulator.

3. Key Contributions

PhysGM Framework: The first feed-forward framework capable of generating a physically-grounded 4D Gaussian simulation from a single image in under one minute, eliminating the need for per-scene optimization.
Novel Training Paradigm: A two-stage approach combining large-scale supervised pre-training with DPO-based refinement. This allows the model to learn from non-differentiable simulators and achieve perceptual realism superior to SDS-based methods.
PhysAssets Dataset: A new benchmark dataset containing 50,000+ 3D assets annotated with physical properties (Young's modulus, Poisson's ratio, material class) and corresponding reference simulation videos. This dataset was constructed using an automated pipeline leveraging Multimodal LLMs (Qwen3-VL) and physics simulators.
State-of-the-Art Performance: Demonstrates that feed-forward inference can achieve higher visual and physical fidelity than optimization-based baselines while being orders of magnitude faster.

4. Experimental Results

Speed: PhysGM generates 4D simulations in < 1 minute (inference time < 30s), compared to baselines like OmniPhysGS (>12 hours) and DreamerPhysics (>0.5 hours).
Quality Metrics:
- CLIPsim: PhysGM with DPO achieved an average score of 0.2748, outperforming baselines (OmniPhysGS: 0.2091, DreamerPhysics: 0.2291).
- User Preference Rate (UPR): In a 4-alternative forced-choice study, PhysGM (w/ DPO) was selected as the most realistic 42.8% of the time, significantly beating baselines (10-17%).
Generalization: The model successfully generalizes to diverse materials (metal, jelly, plasticine, snow, sand) and complex interactions (stretching, dropping, multi-object collisions) from single-view inputs.
Ablation: The DPO stage was shown to be critical, converting a statistically sound prior into a perceptually superior generator.

5. Significance and Impact

PhysGM represents a paradigm shift in 4D content generation:

From Optimization to Inference: It moves the field away from slow, iterative per-scene optimization toward fast, amortized feed-forward inference, making real-time physics-based 4D synthesis feasible.
Unified Representation: By jointly predicting geometry and physics, it captures the intrinsic relationship between appearance and material properties, leading to more realistic simulations.
Scalability: The release of the PhysAssets dataset and the efficiency of the model pave the way for scalable applications in embodied AI, autonomous driving, robotics simulation, and interactive virtual reality, where rapid generation of physically consistent environments is crucial.

The paper concludes that while challenges like the sim-to-real gap and complex fluid dynamics remain, PhysGM sets a new standard for efficiency and quality in physics-grounded 4D synthesis.