PhyCo: Learning Controllable Physical Priors for Generative Motion

1. Problem Statement

Modern video diffusion models have achieved remarkable success in synthesizing realistic textures, lighting, and motion continuity. However, they struggle significantly with physical consistency. Common errors include:

• Objects drifting or falling unrealistically fast under gravity.
• Collisions without realistic bounce (restitution).
• Soft objects failing to deform according to material properties.
• Lack of controllability: Although foundational models are trained on massive datasets, they cannot be controllably manipulated to generate specific variations of physical properties (e.g., changing friction or force magnitude) without retraining or complex external guidance.

Existing solutions fall into two categories, both with limitations:

1. Explicit Simulation: Methods coupling diffusion models with physics solvers (e.g., rigid body dynamics, MPM) require 3D geometry recovery or predefined materials for inference, limiting scalability and generalization.
2. Implicit Priors: Methods using language or trajectory prompts improve semantic consistency but offer no continuous, fine-grained control over diverse physical attributes such as deformation, restitution, and friction.

2. Methodology

PhyCo introduces a framework enabling continuous, interpretable, and physics-grounded control of video generation without requiring simulators or geometry recovery during inference. The approach consists of three core components:

A. Large Physics-Grounded Dataset

• Scale & Diversity: The authors created a dataset with over 100,000 photorealistic simulation videos using the Kubric framework, PyBullet (physics), and Blender (rendering).
• Scenarios: The dataset covers six controlled scenarios (e.g., sliding bricks, bouncing balls, soft body impacts), ranging from rigid collisions to deformable dynamics.
• Annotations: Unlike previous datasets, PhyCo provides pixel-perfect physical property maps for four key attributes:
  1. Friction ($\mu_f$)
  2. Restitution ($e$)
  3. Deformation (Neo-Hookean parameters: $\mu, \lambda, \gamma$)
  4. Applied Force (magnitude and direction)
• Design: The simulations are visually clean but physically rich, allowing the model to decouple visual appearance from underlying dynamics.

B. Physics-Supervised Fine-Tuning (Phase 1)

• Architecture: The authors fine-tune a pre-trained diffusion backbone (Cosmos-Predict2-2B) using a ControlNet architecture.
• Conditioning: The denoising process is conditioned on spatially aligned physical property maps ($p$). These maps are tokenized and projected via an adapter network to align with the DiT backbone (Diffusion Transformer).
• Training Strategy: Only the ControlNet layers are fine-tuned; the weights of the base model and tokenizer remain frozen to preserve pre-trained representations. The model is trained on subsets of the dataset where specific physical properties manifest in the dynamics.

C. VLM-Guided Reward Optimization (Phase 2)

• Motivation: Pure supervised fine-tuning may not guarantee high control precision. To bridge this, a Vision-Language Model (VLM) is used as a differentiable reward signal.
• Process:
  1. Rollout: Instead of evaluating noisy intermediate steps, the model performs an $N$-step denoising rollout to generate a predicted video $\hat{x}_0$.
  2. Evaluation: A fine-tuned VLM (Qwen2.5-VL-3B) analyzes the generated video using a curated "Physics Question Bank." Queries are binary (Yes/No) or regression-based (e.g., "Is the force magnitude between 0.2 and 0.4?" or "Is the movement within the blue sector?").
  3. Reward Signal: The VLM's logits for correct versus incorrect answers are converted into a differentiable reward loss ($L_{VLM}$).
  4. Optimization: The ControlNet is further fine-tuned using this reward loss to align the generated dynamics with the intended physical properties.

3. Main Contributions

1. PhyCo Dataset: A novel, large-scale dataset (over 100,000 videos) with continuous annotations for friction, restitution, deformation, and force, designed to decouple visual appearance from physical dynamics.
2. Controlled Physical Conditioning: A method to feed spatially aligned physical property maps into a diffusion model via ControlNet, enabling continuous control over multiple physical attributes simultaneously.
3. VLM-Guided Alignment: A novel training phase where a fine-tuned VLM provides differentiable feedback on physical consistency, significantly improving control precision without requiring explicit physics solvers during inference.
4. Generalization: The framework demonstrates strong compositional generalization, generating physically consistent motions in stylized or real scenes (e.g., a person jumping on a trampoline), despite being trained exclusively on synthetic data.

4. Experimental Results

The authors evaluated PhyCo on the Physics-IQ Benchmark and through human preference studies.

• Physics-IQ Benchmark:

  • PhyCo significantly outperformed strong baselines (SVD-XT, LTX-Video, CogVideoX, VLIPP) across five domains (solid mechanics, fluid dynamics, optics, magnetism, and thermodynamics).
  • Score: The full PhyCo model (ControlNet + VLM loss) achieved an IQ Score of 43.6 (compared to 34.6 for VLIPP and 29.7 for SG-I2V).
  • Generalization: Even when tested on 120-frame sequences (longer than the 57-frame training data), the model maintained its superior physical realism.

• User Studies (2AFC):

  • In pairwise comparisons, human evaluators preferred PhyCo over baselines for physical realism and controllability.
  • PhyCo showed preference rates of 90.9% for friction, 93.2% for restitution, and 91.3% for deformation compared to the base model Cosmos.

• Ablation Studies:

  • VLM Impact: Adding VLM reward optimization reduced the angular deviation of force direction from approximately 38° (ControlNet only) to 22.5°.
  • Force Direction: PhyCo achieved a mean directional error of 15.2°, significantly better than force prompting (40.5°).
  • Compositional Control: The model successfully handled combinations of attributes (e.g., force + friction) and generalized to unseen objects and scenarios.

5. Significance

PhyCo represents a scalable path toward physically consistent generative world models. Its significance lies in:

• Inference Efficiency: It eliminates the need for explicit physics solvers or 3D reconstruction during inference, making it faster and more flexible than hybrid simulation approaches.
• Interpretability: By using explicit physical property maps, the control is interpretable and aligns with standard parameters of physical simulations.
• Scalability: The method generalizes from synthetic training data to real-world scenarios and diverse object categories, suggesting that embedding physical priors is a viable strategy for building intelligent visual models that understand the laws of physics.

In summary, PhyCo successfully bridges the gap between visual realism and physical plausibility, offering a controllable framework for generating videos that obey the laws of physics through continuous attribute manipulation.