💻 computer science

Enhancing Video Physical Consistency via Role-aware Joint Training and Modality-decoupled Denoising

The paper proposes VPT, a fine-tuning framework for video diffusion models that enhances long-range physical consistency by introducing role-aware signal grouping and a modality-decoupled denoising strategy to mitigate capacity conflicts and inference errors while preserving visual fidelity.

Original authors: Guangting Zheng, Haojing Chen, Hao Li, Jingtao Zhang, Zhen Yang, Xiaosong Jia, Xue Yang, Shaofeng Zhang, Yanyong Zhang

Published 2026-07-09

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Guangting Zheng, Haojing Chen, Hao Li, Jingtao Zhang, Zhen Yang, Xiaosong Jia, Xue Yang, Shaofeng Zhang, Yanyong Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Videos That Look Good but Feel "Wrong"

Imagine you are watching a movie where a character pours wine into a glass. The visuals are stunning—the glass looks real, the wine is red, and the lighting is perfect. But then, something weird happens: the wine defies gravity, floating upward into the glass, or it splashes through the glass as if it were made of ghostly mist.

This is the current state of many AI video generators. They are amazing at painting pretty pictures (visual fidelity), but they often fail at understanding physics (how objects actually move and interact). They don't "know" that a heavy rock falls faster than a feather, or that a liquid can't pass through a solid wall.

The Old Way: Trying to Learn Everything at Once

Previous attempts to fix this tried to teach the AI about physics by showing it "motion maps" (like optical flow, which tracks how pixels move) alongside the video. Think of this like trying to teach a student to drive a car by having them look at the road and a complex map of traffic laws simultaneously, with the teacher shouting instructions for both at the exact same time.

The paper argues this approach has three main flaws:

It treats everyone the same: It doesn't know the difference between the person driving the car (the "agent"), the car itself (the "controlled object"), and a tree on the side of the road (a "passive object"). All of them just get the same generic "move" instruction.
It causes a tug-of-war: The AI gets confused. It tries to make the picture look good and follow the physics map perfectly at the same time. Often, trying to follow the physics map too strictly makes the picture look blurry or weird.
The "Telephone Game" error: During the video creation process, the AI has to guess the motion map for the next step based on its own previous guess. If it makes a tiny mistake early on, that mistake gets amplified with every step, like a game of "Telephone" where the message gets garbled by the end.

The New Solution: VPT (Video Physical-consistency Tuning)

The authors propose a new framework called VPT. Think of VPT as a specialized coaching session for an AI that already knows how to draw, teaching it specifically how to make things move realistically without ruining its drawing skills.

Here are the three main tricks VPT uses:

1. The "Role-Playing" Map (Role-Aware Joint Representation)

Instead of just showing the AI a generic motion map, VPT gives it a "script" that labels every part of the scene with a specific role:

The Agent: The thing doing the action (e.g., a human hand).
The Controlled Object: The thing being moved by the agent (e.g., a baseball glove).
The Passive Object: The thing being hit or affected (e.g., the baseball).
The Background: Everything else (e.g., the sky or the field).

The Analogy: Imagine directing a play. Instead of telling the whole cast "everyone move left," the director says, "The actor playing the hero runs forward, the prop (the ball) flies through the air, and the background scenery stays still." By knowing who is doing what, the AI understands the physics much better.

2. The "Separate Practice" Strategy (Modality-Decoupled Denoising)

In the old method, the AI had to fix the picture and the physics map at the exact same moment. VPT changes the rules: it lets the AI practice fixing the picture and the physics map at different times and different speeds.

The Analogy: Imagine a musician learning a new song.

Old Way: They try to play the melody and the rhythm perfectly at the exact same time, getting frustrated and messing up both.
VPT Way: They practice the rhythm first, then the melody, then combine them. Crucially, they treat the rhythm as a "soft suggestion" rather than a rigid rule. If the rhythm guide is slightly off, the musician doesn't panic; they just use their own good ear (the visual training) to keep the song sounding good. This prevents the AI from "forgetting" how to draw pretty pictures while trying to learn physics.

3. The "Rehearsal" Guide (Cross-step Auto-Guidance)

To stop the "Telephone Game" errors (where small mistakes get huge), VPT uses a clever trick during the video generation. It uses an earlier version of the trained AI as a reference guide for the final AI.

The Analogy: Imagine you are writing a final exam essay. You have a draft you wrote yesterday (the intermediate model) and the final version you are writing now (the final model). Instead of guessing the whole essay from scratch, you look at your draft to see the general direction you were heading, then use that to push your final essay in the right direction without getting stuck on the tiny errors of the draft. This helps the AI stay on the right physical track without getting confused by its own mistakes.

The Results: Better Physics, Same Beauty

The paper tested VPT on existing video models (like Wan2.1).

Before: The AI could make a video of a wine bottle pouring, but the wine might float or splash weirdly.
After (with VPT): The wine pours in a realistic arc, splashes naturally, and interacts correctly with the glass.

The results show that VPT significantly improves the "physical commonsense" of the videos (making them obey the laws of physics) without making the videos look worse or less detailed. It successfully taught the AI to distinguish between a hero, a prop, and a background object, and to learn physics without breaking its artistic skills.

Summary

VPT is like giving a talented artist a new set of instructions: "Don't just paint the scene; understand the roles of the characters, practice the movement separately from the colors, and use your past drafts to guide your final strokes." The result is videos that look beautiful and move like the real world.

Technical Summary: VPT – Enhancing Video Physical Consistency via Role-aware Joint Training and Modality-decoupled Denoising

1. Problem Statement

Modern video diffusion models have achieved remarkable success in generating high-fidelity videos with rich visual details. However, they struggle to maintain long-range physical consistency in complex dynamic scenes. Standard pixel-reconstruction objectives prioritize high-frequency appearance details, often failing to capture the underlying physics of a scene. Consequently, generated videos may exhibit implausible trajectories, unnatural deformations, or inconsistent material interactions (e.g., a wine bottle pouring liquid that defies gravity or deforms unnaturally).

Recent attempts to address this by integrating auxiliary modalities (such as optical flow) via joint training face three critical limitations:

Heterogeneous Dynamics Ignored: Existing methods treat motion uniformly, failing to distinguish the distinct physical roles of different entities (e.g., an active agent, a controlled object, a passive object, and the background).
Capacity Conflict: Jointly modeling visual appearance and auxiliary modalities creates optimization conflicts. As the model learns to fit auxiliary signals, the visual denoising loss often increases, weakening the pretrained visual prior and leading to degraded visual quality or generation collapse.
Inference-Time Error Accumulation: During inference, auxiliary modalities must be predicted recursively by the model itself. Since auxiliary prediction is less mature than visual generation, minor errors in these predictions propagate and amplify across denoising steps, causing unstable motion generation.

2. Methodology: VPT Framework

The authors propose VPT (Video Physical-consistency Tuning), a lightweight fine-tuning framework designed to inject physical priors into pretrained video diffusion models while preserving their visual quality. VPT consists of three core components:

A. Role-Aware Joint Representation

Instead of relying solely on a dense optical flow field, VPT introduces a role-aware map that categorizes scene entities into four distinct physical roles:

Agents: Entities that initiate actions.
Controlled Objects: Entities manipulated by agents.
Passive Objects: Entities affected by external forces.
Background: Static or non-interacting regions.

These roles are encoded as a dense map and concatenated with the optical flow and video latent. This representation allows the model to distinguish entity-specific dynamics and interactions (e.g., an agent actively catching a ball vs. a ball passively reacting to impact) without significant modeling overhead.

B. Modality-Decoupled Denoising with Auxiliary Loss Annealing

To resolve the capacity conflict and dependency issues found in naive joint training, VPT employs two strategies:

Modality-Decoupled Denoising: Unlike previous methods that enforce a shared diffusion timestep for all modalities, VPT assigns independent timesteps ( $t_v, t_f, t_r$ ) to the video, optical flow, and role-aware signals. This decoupling transforms auxiliary modalities from "hard" denoising targets into "soft" physical constraints, allowing the visual branch to maintain its pretrained prior even if auxiliary predictions are imperfect.
Auxiliary Loss Annealing: The weight of the auxiliary losses ( $\lambda$ ) is gradually decayed using a cosine schedule during fine-tuning. Early in training, the model learns motion structure and role-aware biases; later, the focus shifts to preserving the visual prior, preventing the model from over-relying on potentially noisy auxiliary signals.

C. Cross-Step Auto-Guidance

To further enhance physical dynamics without relying on noisy recursive auxiliary predictions during inference, VPT introduces cross-step auto-guidance. This technique uses an intermediate model (from an earlier fine-tuning step) as a reference to guide the final model. By extrapolating the generation direction of the final model relative to the intermediate model, the method amplifies the learned physical dynamics while avoiding the error accumulation associated with standard inner-guidance on auxiliary modalities.

3. Key Contributions

Role-Aware Representation: The introduction of a lightweight semantic anchor that explicitly models the heterogeneous dynamics of agents, controlled objects, passive objects, and backgrounds, addressing the limitation of uniform motion modeling.
Decoupled Training Strategy: A novel training paradigm that separates timesteps and anneals auxiliary losses to mitigate capacity conflicts and prevent the degradation of pretrained visual priors.
Robust Inference Mechanism: The use of cross-step auto-guidance to improve physical consistency without the error accumulation risks of recursively predicted auxiliary modalities.
State-of-the-Art Performance: VPT achieves significant improvements on physical consistency benchmarks while maintaining competitive visual quality, demonstrating that physical priors can be effectively injected via fine-tuning rather than full retraining.

4. Experimental Results

The authors evaluated VPT on the VideoPhy and VideoPhy-2 benchmarks (measuring Semantic Adherence [SA] and Physical Commonsense [PC]) and the VBench suite (measuring general video quality).

Physical Consistency: On the VideoPhy benchmark using the Wan2.1-T2V-1.3B backbone, VPT improved SA from 47.7 to 66.5 and PC from 21.2 to 25.0. On VideoPhy-2, it improved SA from 19.3 to 22.5 and PC from 53.7 to 55.1. Similar gains were observed with the larger 14B backbone.
Visual Quality: Unlike naive joint training or full fine-tuning, which often degrade visual quality, VPT maintained or improved scores on VBench, achieving an overall score of 79.58 (compared to 76.93 for the baseline).
Ablation Studies: Experiments confirmed that:
- Naive joint training leads to visual collapse.
- Role-aware maps provide significant gains over optical flow alone.
- Cross-step auto-guidance yields better physical consistency than inner-guidance with auxiliary modalities.
- The pretrained video VAE can faithfully reconstruct optical flow and role maps, validating the shared latent space approach.

5. Significance and Claims

The paper claims that VPT provides an effective and lightweight solution for enhancing physical consistency in pretrained video diffusion models. By decoupling modalities and explicitly modeling entity roles, VPT successfully injects physics-aware priors without compromising the visual fidelity or prompt adherence of the base model.

The authors emphasize that their approach avoids the pitfalls of previous joint-training methods (capacity conflict and error accumulation) and demonstrates that explicit modeling of physical roles is crucial for realistic interaction dynamics. The work suggests that future video generation models can achieve world-model-like consistency through targeted fine-tuning strategies rather than solely relying on larger-scale pretraining or complex simulation-based pipelines.

Limitations Acknowledged by the Authors:

The framework relies on automatically extracted optical flow and VLM-based role assignments, meaning errors in these upstream tools can affect physical prior learning.
Experiments are currently limited to Wan2.1 backbones and standard benchmarks due to computational constraints.
The scale of existing physical-video datasets limits the ability to explore fine-tuning on substantially larger datasets.