💻 computer science

PhysAgent: Reflective Agentic Physics Control for Physically Plausible Video Generation

PhysAgent introduces a reflective agentic framework that iteratively generates, simulates, verifies, and repairs physics programs to overcome the limitations of one-shot prediction, thereby enabling the creation of more physically plausible and prompt-aligned videos with complex dynamics and interactions.

Original authors: Qirui Li, Jinkun Hao, Yibo Li, Ran Yi, Paul L. Rosin, Yu-Kun Lai

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Qirui Li, Jinkun Hao, Yibo Li, Ran Yi, Paul L. Rosin, Yu-Kun Lai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a movie where a ball rolls off a table, hits a cup, and knocks it over. In real life, physics is the invisible rulebook that makes sure the ball doesn't float away, the cup doesn't turn into jelly, and the timing feels right. For a long time, computers trying to make these videos have been like artists who are great at painting but terrible at math. They could make the ball look like it was rolling, but if you asked them to make it hit the cup at a specific angle or bounce with the right force, they often got it wrong. The ball might slide through the table, or the cup might shatter when it should have just rolled away. This happens because the computer is guessing the whole scene in one big leap, without actually "checking" if the laws of physics hold up before the movie starts.

This paper tackles that problem by introducing a new way to teach computers how to be better directors of physics. Instead of just guessing, the computer now acts like a scientist in a lab. It builds a digital model of the scene, runs a quick test to see what happens, checks if the result matches the user's request, and then fixes its mistakes before making the final video. It's the difference between a chef who throws ingredients into a pot and hopes it tastes good, versus a chef who tastes the soup, realizes it needs more salt, adds a pinch, tastes it again, and then serves it. The goal is to make videos that not only look beautiful but also obey the strict, invisible rules of how the real world moves and crashes.

The "PhysAgent" Detective

The authors of this paper, a team from Shanghai Jiao Tong University and Cardiff University, have built a system they call PhysAgent. Think of PhysAgent as a very persistent, reflective robot director. When you give it a picture (like a bowling alley) and a command (like "make the ball hit the right pin"), it doesn't just immediately start filming. Instead, it enters a loop of try, check, and fix.

Here is how the process works, step-by-step:

The First Guess: The robot looks at your picture and your command. It tries to build a digital version of the scene and writes a "script" for the physics. This script tells the computer things like: "The ball starts at speed 0.5m/s," "The pin is made of wood," and "The force is applied here."
The Simulation (The Test Run): Before making the pretty video, the robot runs a quick, rough simulation. It's like a rehearsal. It watches the digital ball roll and the digital pin fly.
The Reality Check: The robot compares this rehearsal to your original command. Did the ball hit the right pin? Did the stool break when the pumpkin fell on it? If the answer is "no," the robot doesn't just give up. It acts like a detective. It asks, "Why did this fail?"
- Example: If the ball hit the wrong pin, the robot might realize, "Oh, I aimed the force too far to the left."
- Example: If the pumpkin didn't break the stool, it might say, "The stool is too strong, or the pumpkin wasn't heavy enough."
The Fix: The robot then edits its script. It might change the direction of the force to [0.6, 1.0, 0] or increase the speed to 0.8m/s. It doesn't rewrite the whole movie; it only fixes the specific part that went wrong.
Repeat: It runs the test again. If it passes, great! If not, it fixes it again. This cycle can happen up to 10 times until the simulation is perfect.

Why "One-Shot" Guesses Fail

The paper argues strongly against the old way of doing things, which they call "one-shot prediction." In the past, AI models tried to guess all the physics rules in a single instant. The authors found this approach is "brittle," meaning it breaks easily.

Imagine trying to build a house of cards by guessing where every card should go in one second. If you get the first card slightly wrong, the whole tower falls. Similarly, if an AI guesses the weight of an object or the direction of a push incorrectly in its first try, the entire video becomes physically impossible. The paper shows that because all these factors (weight, speed, direction, material) are tightly connected, a small mistake in one area ruins the whole result. PhysAgent solves this by treating the physics script as a "hypothesis" that must be tested and revised, rather than a final answer.

The Magic Toolbelt: Semantic APIs

To make this fixing process easier, the researchers gave the robot a special set of tools called Action APIs. Instead of asking the robot to do complex math like "calculate the friction coefficient of wood on concrete," they gave it simple, human-like commands.

Think of these as the robot's toolbelt. Instead of saying "apply a vector force of 14 Newtons at a 30-degree angle," the robot can just say:

"Set the ball's speed to 0.5m/s."
"Apply a force to the pumpkin."
"Make the stool rigid."
"Fix the cup in place."

This makes the robot much smarter and less likely to make silly math errors. It allows the robot to focus on the story of the physics (e.g., "The ball should roll off and land in the cup") rather than getting stuck on the raw numbers.

What They Found

The team tested PhysAgent on 27 different scenes with 80 different physical events, like a firework curving in the wind or a ring getting stuck on a toy. They compared their method against other top video generators and physics-based AI.

The results were clear:

Better Physics: PhysAgent produced videos that were much more physically plausible. In a test where a judge (an AI) had to decide if the video made sense, PhysAgent scored 0.714 out of 1.0, beating the next best physics-based method (RealWonder) which scored 0.631.
Human Preference: When real people were asked to choose between videos, they preferred PhysAgent's videos 73.4% of the time over the best general video generator (Wan2.2) and up to 93.6% of the time over other physics-based methods.
The Loop Matters: When the researchers removed the "fix-it" loop and let the robot guess only once, the quality dropped significantly. This proved that the ability to check and repair is the secret sauce.

The Bottom Line

This paper suggests that the future of making realistic videos isn't just about making the pictures look prettier; it's about making the computer understand how the world works. By letting the AI run a simulation, check its own work, and fix its mistakes, PhysAgent can create videos where objects collide, break, and bounce exactly as they should. It turns the chaotic process of video generation into a careful, step-by-step experiment, ensuring that when you see a ball hit a cup on your screen, it feels like it really could happen in your living room.

Technical Summary: Reflective Agentic Physics Control for Physically Plausible Video Generation

Problem Statement

Recent advances in video generation have achieved high visual fidelity and temporal coherence, yet they frequently fail to produce physically plausible dynamics. Models often generate videos where objects slide without contact, respond to forces in incorrect directions, or ignore material-dependent behaviors (e.g., rigid bodies breaking or soft objects deforming). This issue is particularly acute in image-to-video generation, where a single input image must be transformed into a plausible future based on a user-specified physical event.

The core challenge lies in fine-grained control over physical dynamics. Existing physics-grounded methods attempt to guide generation by inferring a structured simulator specification (scene state and event parameters) in a single pass using Vision-Language Models (VLMs). However, this approach is inherently brittle because:

Tightly Coupled Factors: The specification involves many interdependent factors (object grounding, material properties, force magnitude, timing). A parameter that seems reasonable in isolation may cause the entire simulation to fail when combined with others.
Execution-Agnostic Prediction: Correctness cannot be determined from the specification alone; it must be evaluated through execution. Single-pass predictions lack the mechanism to detect and repair simulation failures before video synthesis.

Methodology: PhysAgent

The authors propose PhysAgent, a reflective agentic framework that treats physical simulation control as a closed-loop, execution-aware process rather than a one-shot prediction. The framework operates by constructing an executable physical control program, simulating it, verifying the results, and iteratively repairing failures.

1. Framework Architecture

PhysAgent consists of three primary components that operate in a loop:

Program Generator ( $G$ ): Constructs an initial physical control program $P = (S, E)$ $P = (S, E)$ from an input image $I$ $I$ and event prompt $p$ $p$ .
- Scene State ( $S$ ): Defines grounded objects, material types, densities, support geometry, and camera parameters.
- Event Program ( $E$ ): Defines target objects, initial velocities, forces, torques, and timing.
Verifier ( $V$ ): Performs stage-specific progressive validation to isolate errors:
1. Scene Reconstruction Validation: Checks if the 3D scene (geometry, object positions, camera) matches the input image. If this fails, the simulator is not invoked.
2. Physical Simulation Validation: Executes the event program in a physics engine to generate a short diagnostic video. The verifier checks for trajectory correctness, contact timing, force direction, and material response against the prompt.
Planner ( $P$ ): If verification fails, the planner analyzes the diagnosis and memory of previous rounds to generate targeted feedback ( $F$ ). This feedback proposes specific edits (e.g., "increase initial speed," "change force direction," "modify material") rather than regenerating the entire program.

2. Semantic Physics-Control APIs

To reduce the syntactic overhead and brittleness of VLMs directly predicting low-level simulator commands, the authors design a set of semantic Action APIs. The event program is represented as a sequence of high-level operators:

State Initialization: SetVelocity, SetAngularVelocity.
Actuation: ApplyForce, ApplyTorque, ApplyAngledForce.
Disturbance: ApplyDisturbance (e.g., wind).
Constraints: SetPosition, SetOrientation, FixObject.

These APIs allow the agent to reason about physical events at the level of motion and interaction, compiling them into concrete simulator operations only during execution.

3. The Reflective Loop

The process follows an iterative refinement schedule:

Generate: Create an initial program.
Validate: Check scene reconstruction. If passed, run a short diagnostic simulation.
Diagnose: The verifier identifies specific failures (e.g., "ball rolls off table instead of hitting pin").
Repair: The planner updates the program based on the diagnosis (e.g., adjusting force vector from [0.6, 1.0, 0] to [1.4, 1.0, 0]).
Repeat: The loop continues until the diagnostic rollout passes or a maximum iteration budget is reached. The final verified rollout serves as a structured motion prior for the video generator.

Key Contributions

Execution-Aware Formulation: The paper reframes physically plausible video generation as a problem of controlling an executable physical program rather than predicting static parameters. It treats specifications as hypotheses to be tested and revised.
Reflective Agentic Framework: Introduces a closed-loop system that integrates VLM-based generation, physics simulation, reflective verification, and targeted program repair. This enables the system to progressively realize complex trajectories and multi-stage interactions.
Semantic Control Interface: The design of a physics-control API that abstracts low-level numerical commands into expressive, compositional primitives, improving both control flexibility and refinement efficiency.
Progressive Validation: A stage-specific verification strategy that separates reconstruction errors from dynamics errors, preventing unnecessary downstream rollouts and enabling targeted repairs.

Experimental Results

The authors evaluated PhysAgent on a custom benchmark of 27 distinct scenes with 80 event cases (multiple events per scene to test visual consistency).

Quantitative Metrics:
- PhysReal: A normalized GPT-based score for physical plausibility and prompt alignment. PhysAgent achieved 0.714, outperforming general TI2V models (e.g., Wan2.2 at 0.633) and physics-grounded baselines (e.g., PhysCtrl at 0.579, PhysGen3D at 0.480).
- Visual Quality: PhysAgent also achieved higher scores in Visuals (0.644), Aesthetics (0.521), and Consistency (0.266) compared to most baselines, indicating that physical control does not compromise visual fidelity.
User Study (2AFC): In a two-alternative forced-choice study, PhysAgent was preferred over all baselines across three criteria:
- Action Following: 70.2% – 92.4% preference rate.
- Visual Quality: 60.7% – 90.3% preference rate.
- Physical Plausibility: 66.5% – 93.6% preference rate.
Ablation Studies:
- Removing the reflection loop (one-shot prediction) caused the largest drop in PhysReal (0.631), confirming the necessity of iterative repair.
- Replacing semantic APIs with low-level parameters reduced performance and increased the average number of refinement rounds from 3.63 to 5.74, demonstrating that semantic abstractions improve both expressiveness and efficiency.

Significance and Claims

The paper claims that execution-aware physical control is a promising direction for controllable video generation, particularly for prompts requiring precise contact timing, force direction, material response, and multi-stage interactions.

The authors emphasize that their approach moves beyond the limitations of "one-shot" VLM predictions, which often fail to translate user intent into executable simulations due to the complexity of coupled physical factors. By treating physical specifications as executable hypotheses that can be simulated, diagnosed, and repaired, PhysAgent achieves a level of robustness and fine-grained control that existing methods lack. The framework demonstrates that separating scene state from event dynamics and using semantic interfaces allows agents to handle complex physical scenarios without sacrificing visual coherence.