💻 computer science

ProPhy: Progressive Physical Alignment for Dynamic World Simulation

ProPhy is a progressive physical alignment framework that enhances the physical consistency of dynamic world simulations by employing a two-stage Mixture-of-Physics-Experts mechanism to extract fine-grained physical priors and transfer vision-language reasoning capabilities for anisotropic, physics-aware video generation.

Original authors: Zijun Wang, Panwen Hu, Jing Wang, Terry Jingchen Zhang, Yuhao Cheng, Long Chen, Yiqiang Yan, Zutao Jiang, Hanhui Li, Xiaodan Liang

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zijun Wang, Panwen Hu, Jing Wang, Terry Jingchen Zhang, Yuhao Cheng, Long Chen, Yiqiang Yan, Zutao Jiang, Hanhui Li, Xiaodan Liang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are asking a talented but slightly naive artist to paint a scene based on your description: "A basketball hits the ground, kicking up dust, while a cup of coffee sits nearby, steaming in the cold air."

The Problem with Current AI:
Most current video-generating AIs are like that artist who is great at painting colors and shapes but doesn't quite understand how the real world works.

If you ask for the basketball, they might paint the dust rising before the ball hits the ground.
If you ask for the coffee, they might make the liquid level rise magically because of the fire, or make the coffee pot catch fire instantly.
They treat the whole video like a single, blurry blob. They don't realize that the "dust rule" only applies to the basketball, and the "liquid rule" only applies to the coffee. They try to apply one set of "physics rules" to the entire picture at once, leading to chaos.

The Solution: ProPhy (The "Progressive Physical Alignment" Framework)
The researchers behind ProPhy decided to fix this by giving the AI a team of specialized experts and a strict training regimen. Think of ProPhy not as a single artist, but as a high-end film production studio with a very specific workflow.

Here is how ProPhy works, broken down into simple steps:

1. The "Script Readers" (Semantic Experts)

First, before the camera even starts rolling, ProPhy has a team of "Script Readers."

What they do: They read your text prompt ("basketball," "coffee," "snow") and immediately figure out the big picture physics.
The Analogy: Imagine a director looking at a script and saying, "Okay, this scene needs a Gravity Expert for the falling ball and a Fluid Dynamics Expert for the coffee. Let's get those specialists ready."
The Innovation: Unlike older models that just guess the physics, ProPhy explicitly asks, "What kind of physics are we dealing with here?" and assigns the right "expert" to the job.

2. The "Special Effects Crew" (Refinement Experts)

Once the big picture is set, the real magic happens. ProPhy doesn't just apply the "Gravity Expert" to the whole screen. It breaks the video down into tiny pieces (like pixels or small tiles) and assigns a specific expert to each piece.

What they do: They look at the basketball and say, "You need to bounce and kick up dust." They look at the coffee and say, "You need to stay liquid and not catch fire."
The Analogy: Imagine a massive construction site. Instead of one foreman telling everyone to "build a wall," you have a Carpenter working on the wood, a Plumber working on the pipes, and an Electrician on the wires. They all work on their specific parts of the building simultaneously, ensuring the wood doesn't turn into water and the pipes don't catch fire.
The Result: The dust only rises where the ball hits. The coffee stays safe. The physics are "fine-grained" and precise.

3. The "Reality Check" (VLM Alignment)

How do we teach the AI to be this precise? The researchers used a clever trick involving Vision-Language Models (VLMs).

The Problem: The video generator is bad at pointing out exactly where a physical event happens.
The Solution: They used a different type of AI (a VLM) that is really good at looking at a video and describing it. They asked the VLM: "Where exactly is the dust rising?" and "Where is the coffee?"
The Analogy: Think of the VLM as a strict physics teacher. The video generator is the student. The teacher points at the video and says, "No, the dust is here, not there. The coffee is here, not there." The student (ProPhy) then copies the teacher's notes to learn exactly where to apply the rules.

Why This Matters (The "World Simulator")

Current AI is like a dream. In a dream, you can fly, water can turn into fire, and gravity can be optional. It looks pretty, but it's not real.

ProPhy is trying to turn that dream into a simulation.

Old AI: "Here is a video of a ball. It looks like a ball." (But it might float away).
ProPhy: "Here is a video of a ball. It hits the ground, obeys gravity, kicks up dust, and stops. It behaves exactly like a real ball would."

Summary

ProPhy is a new way to teach AI how to make videos that obey the laws of physics. Instead of guessing, it:

Identifies the specific physics rules needed (Gravity, Fluids, Fire).
Assigns a specialist to handle those rules for every tiny part of the video.
Trains using a "teacher" AI that points out exactly where things happen, ensuring the dust rises only when the ball hits, and the coffee doesn't spontaneously combust.

It's the difference between a magic trick and a science experiment. ProPhy wants to build a World Simulator where the video generation is as reliable as reality itself.

1. Problem Statement

Current video generation models, despite achieving high visual realism, struggle to function as true "world simulators" because they fail to adhere to fundamental physical laws. Existing approaches suffer from two primary limitations:

Implicit Physical Guidance: Most models rely on internalizing physical priors during training without explicit guidance during inference. This leads to violations of physics (e.g., objects defying gravity, incorrect fluid dynamics) in complex scenarios.
Lack of Fine-Grained Alignment: Existing methods that do attempt to inject physics (e.g., WISA) often operate at a video-level or sample-level. They assign coarse physical categories to entire videos, failing to capture localized, fine-grained physical phenomena. For instance, in a scene with both fire and water, a video-level model might apply "combustion" logic globally, causing water to evaporate incorrectly or fire to behave like liquid.

2. Methodology: ProPhy Framework

ProPhy introduces a Progressive Physical Alignment Framework designed to inject explicit, learnable physics priors and refine them through a two-stage Mixture-of-Physics-Experts (MoPE) mechanism. The framework operates on top of latent video diffusion backbones (e.g., Wan2.1, CogVideoX).

A. Two-Stage Mixture-of-Physics-Experts (MoPE)

Instead of a single uniform architecture, ProPhy employs a hierarchical expert system:

Semantic Expert Block (SEB) - Video Level:
- Function: Extracts global physical priors from the text prompt.
- Mechanism: Contains learnable physical basis maps (experts) representing distinct physical concepts (e.g., combustion, fluid dynamics). A Semantic Router analyzes the text embedding and dynamically weights these experts to produce a video-level physical prior.
- Goal: To establish a discriminative, high-level understanding of the physical laws governing the scene.
Refinement Expert Block (REB) - Token Level:
- Function: Refines the global prior into fine-grained, token-level physical dynamics.
- Mechanism: Operates on individual latent tokens. A Refinement Router predicts the specific physical law relevant to each token's spatial location and selects the top- $k$ experts to process that token.
- Goal: To enable anisotropic generation, where different spatial regions respond accurately to localized physical cues (e.g., dust rising only where a ball hits the ground, not globally).

B. Physical Alignment Strategy

To train the Refinement Experts to understand spatial localization, ProPhy transfers physical reasoning capabilities from Vision-Language Models (VLMs) (specifically Qwen2.5-VL) to the generative model.

VLM Supervision: The VLM is prompted to describe physical phenomena in a video. By analyzing the attention maps between the VLM's text tokens (describing the phenomenon) and visual tokens, the system generates a "ground truth" localization map for physical events.
Alignment Loss: The Refinement Router's output is aligned with these VLM-derived attention maps using a Fine-grained Alignment Loss ( $L_{fine-align}$ ). This forces the generative model to learn exactly where specific physical laws apply within the video frame.

C. Training Objectives

The total loss function combines standard diffusion loss with three specific alignment terms:

$L_{coarse}$ (Semantic Alignment): Ensures samples with the same physical category share similar routing weights in the SEB.
$L_{fine-align}$ (Token Alignment): Aligns the token-level router outputs with VLM-derived spatial attention maps.
$L_{fine-balance}$ (Load Balancing): Prevents mode collapse by ensuring all refinement experts are utilized fairly.

3. Key Contributions

Progressive Physical Alignment: A novel framework that moves from coarse, video-level physical understanding to fine-grained, token-level physical reasoning, enabling spatially anisotropic generation.
Two-Stage MoPE Mechanism: A specialized architecture with Semantic and Refinement Experts that allows the model to learn discriminative physical representations and capture complex, co-existing physical phenomena.
VLM-Guided Fine-Grained Alignment: An innovative strategy that leverages the superior spatial localization of VLMs to supervise the generative model's token-level physical reasoning, solving the problem of coarse global guidance.
State-of-the-Art Performance: Extensive experiments demonstrate that ProPhy significantly outperforms existing methods in maintaining physical consistency without sacrificing visual quality.

4. Experimental Results

The authors evaluated ProPhy on the VideoPhy2 benchmark (focusing on physical commonsense and semantic adherence) and VBench (visual quality).

Quantitative Performance:
- On the VideoPhy2 benchmark, ProPhy applied to CogVideoX-5B achieved a Joint Pass Rate of 26.7%, a substantial improvement over the base model (22.3%) and other physics-aware methods like WISA (25.8%) and VideoREPA (22.0%).
- It showed significant gains in the HARD subset (challenging scenarios), with a +19.7% improvement in the Joint metric over the base model.
- On VBench, ProPhy improved the Dynamic Degree score significantly (e.g., 72.0 vs. 46.8 for the base CogVideoX), indicating better capture of dynamic physical behaviors, while maintaining high aesthetic and imaging quality.
Qualitative Analysis:
- Discus Throw: ProPhy correctly localized dust emission only at the point of ground contact, whereas prior models generated dust globally or coupled it incorrectly with the trajectory.
- Collisions: ProPhy adhered to momentum conservation (e.g., a large ball transferring energy to a small one), while others showed penetration artifacts or incorrect motion.
- Fluid Dynamics: In scenarios involving coffee pouring near a fire, ProPhy correctly kept the liquid level stable and prevented the pot from catching fire, whereas baselines showed liquid rising abnormally or fire spreading to the pot.
Ablation Studies:
- Removing the Physical Branch or using only LoRA fine-tuning resulted in significantly lower performance.
- Both the Semantic Expert Block (SEB) and Refinement Expert Block (REB) were necessary; removing either degraded performance.
- The specific alignment losses (relative vs. absolute) were critical; the proposed relative loss formulation provided better guidance than standard BCE.

5. Significance

ProPhy represents a significant step forward in Video Generation as World Simulation. By explicitly modeling physical priors and aligning them at a fine-grained token level, it bridges the gap between visual realism and physical plausibility.

Scientific Impact: It demonstrates that injecting structured, localized physical knowledge via VLM supervision is more effective than relying solely on implicit learning from massive datasets.
Application Potential: The ability to generate physically consistent, dynamic videos is crucial for applications in robotic simulation, autonomous driving training, and scientific visualization, where logical consistency is as important as visual fidelity.
Future Direction: The paper suggests that future work could integrate these aligned regions with governing differential equations to inject even more interpretable and principled physical knowledge.