💻 computer science

PhysVideo: Physically Plausible Video Generation with Cross-View Geometry Guidance

PhysVideo introduces a two-stage framework that leverages a newly constructed multi-view dataset (PhysMV) and cross-view geometry guidance to generate physically plausible and spatially coherent videos by first synthesizing physics-aware orthogonal foregrounds and then integrating them with background contexts.

Original authors: Cong Wang, Hanxin Zhu, Xiao Tang, Jiayi Luo, Xin Jin, Long Chen, Fei-Yue Wang, Zhibo Chen

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Cong Wang, Hanxin Zhu, Xiao Tang, Jiayi Luo, Xin Jin, Long Chen, Fei-Yue Wang, Zhibo Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to make a movie. You give it a picture of a toy pig and say, "Make it fall and bounce."

Most current AI video generators are like talented but clumsy painters. They are great at copying the look of things (the colors, the textures), but they don't really understand how things move in the real world. If you ask them to make a heavy rock fall, it might float like a feather. If you ask for a rubber ball to bounce, it might shatter like glass. They are guessing the motion based on patterns they've seen, not based on the laws of physics.

PhysVideo is a new approach that acts like a smart director who hires a physics expert. Instead of just guessing, it breaks the movie-making process into two distinct steps to ensure everything feels "real."

Here is how it works, using a simple analogy:

The Problem: The "Flat" Illusion

Real life happens in 3D. When a ball bounces, it moves in a specific way that looks consistent whether you are watching it from the front, the side, or the back.
Current AI models usually try to generate the video from just one angle (like looking through a camera lens). Because they don't "see" the 3D shape, they often mess up the physics. The ball might warp, or the bounce might look weird when the camera angle changes.

The Solution: The "Two-Stage" Factory

PhysVideo solves this by building the video in two stages, like a specialized animation studio.

Stage 1: The "Physics Lab" (Phys4View)

Before making the final movie, the AI goes into a "Physics Lab."

The Setup: You give it the picture of the object, a text description, and specific "physics rules" (e.g., "This is a heavy metal ball," or "This is a soft rubber duck").
The Magic Trick: Instead of making just one video, this stage generates four videos at the same time, showing the object from four different angles (Front, Back, Left, Right) simultaneously.
The "Cross-View" Glue: Imagine four friends standing in a circle, all drawing the same falling ball. If one friend draws the ball moving left, the others must also draw it moving left. PhysVideo uses a special "glue" (called Cross-View Geometry Guidance) to make sure all four drawings stay perfectly synchronized. If the ball bounces in the front view, it bounces in the side view too.
The Result: You now have a set of "orthogonal videos" (four views) where the object moves perfectly according to the laws of physics (gravity, elasticity, density).

Stage 2: The "Movie Studio" (VideoSyn)

Now that the physics expert has created the perfect, synchronized movement of the object, the second stage takes over.

The Task: This stage is the "Artist." Its job is to take the moving object from Stage 1 and paint a beautiful background around it (like a room, a street, or a park).
The Guidance: It doesn't guess how the object moves. It uses the "Physics Lab" videos as a strict guide. It knows exactly where the object is and how it's moving, so it can paint the background to interact with it realistically (e.g., the object casting a shadow, or bouncing off the floor).
The Result: A full, high-quality video where the object moves with perfect physical realism, and the background looks natural.

Why is this a big deal?

Think of it like the difference between drawing a cartoon and filming a real scene.

Old AI: Like drawing a cartoon. It looks good, but if you look closely, the physics are "cartoonish" (things float, bounce weirdly, or change shape).
PhysVideo: Like filming a real scene with a real object. Because it generated the motion from four angles simultaneously using physics rules, the movement is consistent no matter how you look at it.

The "Training Gym" (PhysMV Dataset)

To teach the AI how to do this, the researchers built a massive "gym" called PhysMV.

They used a physics engine (like a video game simulator) to create 40,000 different scenes.
For every scene, they recorded the object falling, bouncing, or rolling from four different angles at the same time.
This gave the AI 160,000 video clips to study, teaching it exactly how a heavy metal ball moves differently from a soft rubber duck, and how those movements look from every angle.

Summary

PhysVideo is like giving the AI a physics textbook and a multi-camera rig.

It first simulates the object's movement from four angles to ensure the physics are 100% correct.
Then, it uses that perfect movement to generate a beautiful, realistic video with a background.

The result? Videos where objects don't just look real; they act real. They bounce, fall, and collide exactly the way they would in our physical world.

1. Problem Statement

Current video generation models, despite achieving high visual fidelity, struggle to produce physically plausible motion.

The Core Limitation: Real-world object motion occurs in 3D space governed by physical laws (mass, elasticity, gravity), but standard video generators only learn 2D pixel-space appearance transformations.
Consequences: Generated videos often exhibit implausible accelerations, inconsistent object interactions, and motion patterns that ignore intrinsic object properties (e.g., a heavy object bouncing like a rubber ball).
Existing Solutions' Flaws:
- Data-driven models: Lack physical constraints.
- Physics-engine-based methods: Rely on explicit 3D representations and multi-stage pipelines (e.g., MPM solvers, 3D-GS) that are difficult to scale, hard to integrate into end-to-end training, and struggle with flexible multimodal conditioning (text/image to video).

2. Methodology: PhysVideo Framework

The authors propose PhysVideo, a two-stage, feed-forward framework that generates physically plausible videos without explicitly reconstructing 3D geometry during inference. It relies on cross-view geometry guidance to enforce physical consistency.

Stage 1: Phys4View (Physics-Aware Multi-View Foreground Generation)

This stage generates synchronized videos of the foreground object from four orthogonal viewpoints, conditioned on an input image, text prompt, and explicit physical attributes.

Inputs: Single image ( $I$ ), text prompt ( $T$ ), and physical attributes ( $P$ ) including motion cues (acceleration, velocity) and material properties (density, Young's modulus, Poisson's ratio).
Video Priors:
- Foreground Initialization: Creates a static foreground video from the input mask.
- Structural Prior: Uses a pretrained image-to-3D model to generate a static multi-view video for geometric guidance.
Key Architectural Innovations:
1. Physics-Aware Attention: Injects physical parameters directly into the attention mechanism to modulate motion dynamics based on material properties.
2. Geometry-Enhanced Cross-View Attention: Instead of explicit 3D modeling, the model predicts a depth map and computes pairwise geometric distances between views. This creates a "geometry bias" in the attention logits, ensuring that corresponding points across different views remain spatially consistent.
3. Temporal Attention: A lightweight module to enhance temporal coherence within each view.
Output: Four synchronized orthogonal videos of the foreground object.

Stage 2: VideoSyn (Background-Aware Video Synthesis)

This stage synthesizes the final full video (foreground + background) using the generated multi-view foreground as guidance.

Process:
1. Extracts one view from the Phys4View output.
2. Uses frame interpolation to densify the temporal resolution.
3. Computes Optical Flow from the interpolated video to capture fine-grained motion dynamics independent of appearance.
Guidance: The optical flow video is used as an explicit motion control condition in the diffusion process, allowing the model to learn realistic foreground-background interactions (occlusion, contact) in a data-driven manner.

3. Key Contributions

PhysVideo Framework: A novel two-stage pipeline that decouples physics-aware foreground motion generation from background synthesis, enabling controllable video generation conditioned on explicit physical attributes.
Phys4View Model: Introduces Physics-Aware Attention and Geometry-Enhanced Cross-View Attention to generate temporally coherent and geometrically consistent multi-view motion without explicit 3D reconstruction.
VideoSyn Model: Enables realistic foreground-background interaction synthesis by leveraging multi-view motion cues (optical flow) as guidance, avoiding the need for explicit physics simulation at inference time.
PhysMV Dataset: A large-scale, physics-oriented multi-view video dataset containing 40,000 scenes (160,000 video sequences) generated using a physics engine and 3D Gaussian Splatting (3D-GS), providing the first large-scale resource for training multi-view physics-aware video models.

4. Experimental Results

The method was evaluated on a benchmark of 33 test samples involving rigid and deformable objects.

Quantitative Performance:
- WorldScore: PhysVideo achieved the highest scores in Photometric Consistency (94.09), Motion Smoothness (93.62), and 3D Consistency (87.19), outperforming state-of-the-art models like CogVideoX, Wan, and physics-based baselines like PhysGen3D.
- VBench: Achieved superior scores in Motion (0.996), Subject Consistency (0.970), and Flickering reduction.
- GPT-4o Evaluation: Scored highest in Physical Realism (0.627), surpassing all competitors, while maintaining high Photorealism (0.885) and Semantic Consistency (0.636).
Qualitative Performance:
- Visual comparisons show that PhysVideo correctly simulates object deformation and motion based on material properties (e.g., a soft ball deforming vs. a hard metal object), whereas baselines often produce texture degradation or physically impossible trajectories.
- The method maintains consistent object identity and motion across different viewpoints.

5. Significance and Impact

Bridging the Gap: PhysVideo successfully bridges the gap between data-driven generative models and physics-based simulation, achieving physical plausibility without the computational overhead of explicit 3D solvers during inference.
Scalability: By avoiding explicit 3D reconstruction pipelines, the framework is more scalable and compatible with flexible multimodal inputs (text, image, physics parameters).
New Benchmark: The introduction of the PhysMV dataset addresses the lack of large-scale, multi-view, physics-grounded video data, facilitating future research in 4D generation and embodied AI.
Applications: The generated multi-view foregrounds enable downstream tasks such as object editing (e.g., rotating an object while maintaining physical consistency) and controllable simulation in dynamic environments.

In conclusion, PhysVideo represents a significant step forward in video generation by embedding physical laws and geometric consistency directly into the generative process through cross-view attention mechanisms, resulting in videos that are not only visually high-fidelity but also physically plausible.