🤖 AI

PhysVid: Physics Aware Local Conditioning for Generative Video Models

PhysVid introduces a physics-aware local conditioning scheme that annotates temporally contiguous frame chunks with grounded descriptions and employs negative physics prompts to significantly enhance the physical plausibility of generative video models.

Original authors: Saurabh, Pathak, Elahe Arani, Mykola Pechenizkiy, Bahram Zonooz

Published 2026-03-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Saurabh, Pathak, Elahe Arani, Mykola Pechenizkiy, Bahram Zonooz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot artist how to paint a moving picture (a video) based on a simple sentence like, "A car drives down a rainy street."

Current AI video generators are like incredibly talented but slightly clumsy artists. They can make the car look beautiful and the rain look shiny, but they often mess up the physics. The car might float like a ghost, the rain might fall upward, or the wheels might spin backward while the car moves forward. They know what things look like, but they don't really understand how things work.

PhysVid is a new method that teaches the AI to be a better "physics student" so it doesn't just guess; it understands the rules of the universe.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Global" vs. The "Local"

Imagine you are directing a movie.

Old Way (Global Prompt): You give the director one big instruction for the whole movie: "Make a movie about a car driving in the rain."
- The Issue: The director gets the general vibe, but when it comes to the specific moment the car hits a puddle, they might forget the rules of water. They might make the water splash upward or the car slide sideways like it's on ice, even if the road is dry. The instruction was too broad to catch the tiny, split-second details.
PhysVid's Way (Local Conditioning): Instead of one big instruction, PhysVid breaks the movie into tiny 1-second clips. For each clip, it gives the director a specific, physics-focused note.
- Clip 1: "The car moves forward; the wheels spin forward."
- Clip 2: "The car hits a puddle; water splashes outward and downward due to gravity."
- Clip 3: "The rain falls straight down; the windshield wipers push water sideways."

By focusing on these tiny chunks, the AI pays attention to the specific rules that apply right now, rather than just the general theme.

2. The Secret Sauce: The "Physics Tutor" (VLM)

How does the AI know what those specific notes should be? It uses a Vision Language Model (VLM), which acts like a Physics Tutor.

During Training: The AI watches thousands of real videos. The Tutor looks at a 1-second clip and says, "Okay, in this specific second, the ball is falling because of gravity, and the shadow is getting longer because the sun is moving." It writes these rules down as a special "physics prompt."
The Result: The AI learns to associate the visual of a falling ball not just with the word "ball," but with the specific rule "gravity pulls it down."

3. The "Anti-Gravity" Trick (Counterfactuals)

This is the cleverest part. To teach the AI what not to do, PhysVid uses a "Devil's Advocate" approach.

The Positive Prompt: "The ball falls down."
The Counterfactual (Negative) Prompt: The AI is also asked to imagine the wrong version: "The ball floats up," or "The ball turns into a square while falling."

During the generation process, the AI is told: "Make the video look like the Positive Prompt, but make sure it looks NOTHING like the Negative Prompt."

Think of it like a teacher correcting a student: "Don't draw the car floating! Draw it on the road!" By explicitly showing the AI what "bad physics" looks like, it learns to steer clear of those mistakes.

4. The Result: A Smarter, Smaller Artist

The paper shows that PhysVid is a 1.7 billion parameter model. For context, the giant model it is compared to (Wan-14B) is 14 billion parameters—almost 10 times bigger.

The Analogy: Imagine a giant encyclopedia (the big model) that knows everything but is slow and sometimes gets confused by the details. PhysVid is like a small, sharp-witted detective who only cares about the rules of physics.
The Outcome: Even though PhysVid is much smaller, it generates videos that obey the laws of physics 33% better than the massive models. The water flows correctly, the cars drive realistically, and objects don't magically disappear or change shape.

Summary

PhysVid is like giving a video generator a magnifying glass and a rulebook. Instead of looking at the whole movie at once, it zooms in on every second, checks the physics rules for that specific moment, and uses a "what-if" game to avoid silly mistakes. The result is a video that feels real because the AI finally understands that gravity pulls things down, not up.

1. Problem Statement

Generative video models (e.g., Sora, Wan) have achieved high visual fidelity but frequently fail to adhere to basic physical laws (e.g., gravity, fluid dynamics, object collisions). This limits their reliability in real-world applications like robotics and simulation.

Limitations of Current Approaches:
- Global Text Conditioning: Standard Text-to-Video (T2V) models rely on a single global prompt. This is often too coarse to capture fine-grained, short-horizon physical interactions that evolve rapidly within specific time intervals.
- Frame-Level Conditioning: Previous attempts to inject physics often rely on frame-level signals that are domain-specific or short-horizon, lacking temporal continuity.
- Global Physics Constraints: Methods that apply physics constraints over the entire video duration fail to capture local, transient physical phenomena.
Core Challenge: How to inject fine-grained, temporally local physics knowledge into generative models without relying on explicit physics simulators or massive, physics-labeled datasets, while maintaining coherence with the global narrative.

2. Methodology: PhysVid

PhysVid introduces a physics-aware local conditioning scheme that operates over temporally contiguous chunks of frames. The approach consists of three main stages:

A. Data Annotation (Training Phase)

Instead of using pre-existing global physics labels, PhysVid generates its own local physics annotations using a Vision Language Model (VLM).

Segmentation: Training videos are divided into fixed-duration temporal chunks (e.g., ~0.7 seconds).
VLM Analysis: A VLM (VideoLLama3-7B) analyzes each chunk individually, guided by the global caption. It identifies visible elements and describes physical phenomena in three domains:
- Dynamics: Motion, forces, energy, momentum.
- Shape: Deformation, elasticity, structural integrity.
- Optics: Lighting, reflections, shadows, refractions.
Prompt Generation: The VLM outputs a structured, concise text prompt for each chunk that describes the physics observed in that specific segment.

B. Model Architecture: Chunk-Aware Cross-Attention

The authors modify a pretrained Diffusion Transformer (based on Wan2.1) to incorporate these local prompts.

Dual Conditioning Pathway: The model accepts both the global text prompt ( $c_g$ ) and a set of local physics prompts ( $C = \{c_1, ..., c_k\}$ ).
Chunk-Aware Cross-Attention:
- Standard T2V models use global cross-attention where all video tokens attend to the same global text tokens.
- PhysVid inserts local cross-attention blocks where video query tokens attend specifically to text keys from their corresponding temporal chunk.
- Rotary Positional Embeddings (RoPE): RoPE is applied to both video and text modalities. Crucially, a 2D grid is defined for text tokens: one dimension for the chunk index and one for the intra-chunk position. This allows the model to distinguish between text describing the current moment versus text describing a different time segment, ensuring precise temporal alignment.
Training: The model is trained using Flow Matching. The base layers are initially frozen while the new attention modules are trained, followed by full fine-tuning.

C. Inference with Counterfactual Guidance

During inference, the video is generated from noise, so the local visual data required for VLM annotation is unavailable.

Local Prompt Synthesis: An LLM generates a sequence of local physics prompts based only on the global text prompt, ensuring temporal coherence and alignment with the global narrative.
Counterfactual Generation: To steer the generation away from physically implausible trajectories, the system generates counterfactual prompts. These describe the same scene but with deliberate violations of physical laws (e.g., "the car floats," "shadows move opposite to the light").
Classifier-Free Guidance (CFG): The generation process uses a modified CFG formula that incorporates both the positive local prompts and the counterfactual negative prompts at both the global and chunk levels:
$x_{T-1} = (1 + w) \cdot G(x_T, c_g, C, T) - w \cdot G(x_T, c_n, C', T)$
Where $C'$ is the set of counterfactual prompts and $w$ is the guidance scale. This effectively pushes the model away from physically incorrect states.

3. Key Contributions

Local Temporal Conditioning: A novel conditioning mechanism that operates on groups of frames (chunks) rather than the whole video or single frames, preserving sufficient temporal context to observe local physical laws.
VLM-Driven Annotation: A pipeline to automatically extract physics-grounded descriptions from video data using VLMs, eliminating the need for manual physics labeling.
Counterfactual Guidance: The introduction of negative physics prompts (counterfactuals) during inference to explicitly penalize physically impossible trajectories, improving physical commonsense without retraining.
Efficiency: The method achieves state-of-the-art physical realism with a significantly smaller model size (1.7B parameters) compared to massive baselines (14B+).

4. Experimental Results

The method was evaluated on VideoPhy and VideoPhy2, benchmarks designed to test physical commonsense in video generation.

Quantitative Performance:
- VideoPhy: PhysVid (1.7B) achieved a Physical Commonsense (PC) score of 0.32, a ~33% relative improvement over the Wan-14B baseline (0.24) and the Wan-1.3B baseline (0.24).
- VideoPhy2: PhysVid achieved a PC score of 0.64, outperforming Wan-14B (0.59) by ~8%.
- Comparison: PhysVid outperforms or matches many larger models (e.g., CogVideoX-5B, HunYuanVideo) and specialized physics-aware methods (e.g., WISA, PhyT2V) despite having fewer parameters.
Qualitative Results:
- Visual comparisons show PhysVid generates more realistic fluid dynamics (pouring wine, waterfalls), object interactions (blenders, skateboards), and lighting/shadow consistency compared to Wan-14B.
- Ablation studies confirm that both the chunk-aware architecture and the counterfactual guidance are critical for the performance gains. Removing counterfactuals reduces the PC score.
Efficiency: While adding VLM overhead, the total inference time per video (93s) is significantly lower than the 14B baseline (310s) while offering superior physical realism.

5. Significance and Impact

Bridging the Gap: PhysVid demonstrates that explicit, localized physics conditioning can overcome the "ceiling" of learning physics purely from data distribution. It proves that smaller models can outperform larger ones in physical plausibility if the conditioning is structurally aligned with physical laws.
Scalability: The approach is generalizable to any dataset because it extracts physics information dynamically via VLMs rather than relying on static, pre-labeled physics datasets.
Real-World Application: By improving physical commonsense, this work moves generative video models closer to being reliable "world simulators," which is crucial for high-stakes domains like autonomous driving, robotics training, and healthcare simulation.
Future Direction: The paper highlights the potential of combining large language models (for reasoning) with diffusion models (for generation) via structured, temporally aware attention mechanisms to solve complex physical reasoning tasks.