Learning to Generate Rigid Body Interactions with Video Diffusion Models

This paper introduces KineMask, a novel video generation framework that leverages a two-stage training strategy with object masks and combined low-level motion and high-level textual conditioning to enable physically plausible rigid body interactions and generalization to real-world scenes.

The Gist

Imagine you have a magical movie camera that can take a single photo and turn it into a video. This is what modern Video Diffusion Models (VDMs) do. They are incredibly talented artists, but they have a major blind spot: they don't really understand physics.

If you ask a standard AI to make a video of a cup sliding across a table and hitting a stack of books, the AI might make the cup pass through the books like a ghost, or make the books fly into space like they were hit by a rocket. It's creative, but it's not real.

Enter KineMask, a new "training program" for these AI cameras that teaches them how to be physics professors.

Here is how KineMask works, explained through simple analogies:

1. The Problem: The "Ghost" Camera

Current AI video generators are like actors who have memorized a script but don't understand the plot. They know what a "collision" looks like in a picture, but they don't understand the cause and effect. If you push a ball, they don't know it should hit the wall and bounce back; they might just make the ball disappear or turn into a flower.

2. The Solution: The "Training Wheels" (Two-Stage Training)

KineMask teaches the AI using a clever two-step process, similar to how a child learns to ride a bike.

• Stage 1: The Training Wheels (Full Guidance)
  First, the AI is shown thousands of videos made in a computer simulator (like a video game). In these videos, the AI is given a "training mask" for every single frame. This mask is like a glowing outline that says, "Hey, this object is moving at this speed right now." The AI learns to copy these movements perfectly. It's like riding a bike with training wheels; the AI knows exactly where everything is supposed to go.

• Stage 2: Taking the Training Wheels Off (The Magic Trick)
  Here is the genius part. The researchers start hiding the training masks. They show the AI the first frame with the mask (the push), but then they cover up the masks for the rest of the video.

  • The Challenge: The AI has to guess what happens next. "I see the cup moving left... okay, based on what I learned in Stage 1, it must hit the wall, stop, and maybe knock over a book."
  • The Result: The AI learns to predict the future. It stops copying and starts reasoning. It learns that if Object A hits Object B, Object B must move.

3. The Secret Sauce: The "Scriptwriter" (Text Conditioning)

KineMask doesn't just use math; it uses a "Scriptwriter" (a language AI).

• The Low-Level Control: You draw an arrow on a photo to say, "Move this cup here." This is the physical instruction.
• The High-Level Control: The system asks a language AI to write a short story about what should happen. "The cup slides, hits the wall, and shatters into pieces."
• The Combination: The video AI uses the arrow to know where to move, and the story to know how it should look when it breaks. This allows it to create complex effects like water splashing or glass shattering, which are usually impossible for these models.

4. The Result: A World Simulator

Once trained, KineMask can take a real photo from your phone (like a messy kitchen counter) and let you interact with it.

• You say: "Push this coffee cup to the right."
• KineMask generates: A video where the cup slides, hits a stack of plates, and the plates topple over realistically.
• Why it matters: This isn't just for making cool movies. This is the first step toward robots that can learn by watching videos. If a robot can simulate a collision in a video before trying it in real life, it won't break your expensive vase.

Summary Analogy

Think of standard AI video generators as improvisational comedians. They are funny and creative, but if you ask them to act out a car crash, they might make the cars turn into butterflies.

KineMask is like hiring a stunt coordinator and a physics teacher to train those comedians.

1. They practice on a safe, fake set (the simulator).
2. They are forced to predict the outcome without seeing the script (the mask dropout).
3. They are given a script that describes the physics (the text prompt).

Now, when you ask them to act out a crash, they don't turn cars into butterflies. They make the cars crash, bounce, and scatter debris exactly the way physics demands.

Technical Summary

1. Problem Statement

Recent Video Diffusion Models (VDMs) have achieved high visual quality and temporal consistency, making them promising candidates for "world simulators" in robotics and embodied AI. However, current models suffer from significant limitations:

• Lack of Physical Plausibility: They struggle to simulate fundamental physical traits such as object permanence, collisions, and causal interactions. Objects often fly, disappear, or pass through each other.
• Limited Control Mechanisms: Existing control methods (e.g., drag-based approaches) require pre-defined target trajectories or full motion paths, preventing the inference of causal effects from initial conditions alone.
• Simulation Dependency: Physics-aware approaches often rely on explicit 3D scene reconstruction or external physics simulators, which are computationally expensive and limit flexibility.

The core challenge is to enable VDMs to generate realistic rigid body interactions given only an initial image and the initial velocity of an object, without requiring pre-defined future trajectories or explicit 3D reconstruction.

2. Methodology: KineMask

The authors propose KineMask, a framework that integrates low-level kinematic control with high-level textual conditioning into Video Diffusion Models.

A. Conditioning Mechanism

KineMask introduces a novel control signal based on object masks encoding velocity:

• Input: A single reference image ($y$) and a velocity mask ($m$).
• Velocity Mask: A dense mask where the RGB channels encode the instantaneous velocity vector ($x, y, z$) of specific objects in the first frame.
• Architecture: The method employs a ControlNet branch ($\psi_\phi$) attached to a pre-trained VDM (e.g., CogVideoX). This branch processes the velocity mask to guide the denoising process.

B. Two-Stage Training Strategy

To teach the model to infer future interactions from only initial conditions, the authors propose a two-stage training pipeline using synthetic data generated in Blender:

1. Stage 1 (Full Supervision): The model is trained with velocity masks provided for every frame of the video sequence. This allows the ControlNet to learn the mapping between pixel-wise motion supervision and the resulting video dynamics.
2. Stage 2 (Mask Dropout / Inference Alignment): To simulate the real-world inference scenario (where only the initial velocity is known), the training strategy is modified. During training, velocity masks for frames $f > 0$ are randomly dropped (set to zero). The model is forced to predict the subsequent motion and interactions based only on the initial frame's velocity mask ($m_0$) and the learned physical priors.

C. High-Level Textual Conditioning

To enhance the realism of complex effects (e.g., liquid spilling, glass shattering), KineMask integrates a high-level textual description ($c$):

• Training: A Vision-Language Model (VLM) generates detailed captions describing the physical interactions in the synthetic videos.
• Inference: A Large Language Model (LLM, e.g., GPT-5) predicts a description of the scene's future dynamics based on the user's input direction and velocity.
• Integration: Both the velocity mask (low-level) and the text prompt (high-level) condition the VDM simultaneously.

3. Key Contributions

1. KineMask Framework: A novel mechanism for object motion conditioning in VDMs using a velocity-encoded mask and a two-stage training strategy.
2. Synthetic-to-Real Generalization: The method is trained on simple synthetic scenes (cubes/cylinders) but successfully generalizes to complex real-world scenes, demonstrating that VDMs can learn physical laws from synthetic data.
3. Causal Interaction Generation: Unlike drag-based methods, KineMask infers causal effects (collisions, pushing, knocking over) solely from initial velocity, enabling the generation of complex dynamical phenomena.
4. Generalization Across Models: The approach is model-agnostic and has been validated on multiple VDM backbones (CogVideoX-5B, Wan2.2-5B, Cosmos2.5-2B), consistently outperforming their original versions.

4. Experimental Results

The authors evaluated KineMask on synthetic test sets and a "Real World" dataset of 50 images.

• Quantitative Performance:
  • Metrics: Outperformed baselines (CogVideoX, Wan, TORA, MotionI2V, Force Prompting) in Fréchet Video Distance (FVD), Mean Squared Error (MSE), and Fréchet Video Motion Distance (FVMD).
  • Object Consistency: Achieved higher Intersection over Union (IoU) scores, indicating better geometric consistency of moving objects.
• Qualitative Improvements:
  • Realism: Successfully generated realistic collisions, liquid splashes, and object shattering.
  • Failure Modes: Baselines often produced hallucinations (objects flying, disappearing) or ignored collisions. KineMask maintained physical plausibility.
• User Study: In a pairwise comparison with 30 participants, KineMask was preferred over all baselines in Motion Fidelity, Interaction Realism, and Physical Consistency (winning rates ranged from 68% to 88%).
• Ablation Studies:
  • Two-Stage Training: Essential for performance; training without the dropout strategy failed to generalize to single-frame inputs.
  • Data Influence: Training on "Interactions" data (with collisions) was crucial; training only on "Simple Motion" (no collisions) led to hallucinations when collisions were expected.
  • Text Conditioning: High-level text prompts significantly improved the generation of complex effects (e.g., breaking glass) that were not explicitly present in the training distribution.

5. Significance and Future Work

• World Modeling: KineMask represents a significant step toward reliable world models for robotics and embodied decision-making, where understanding causal physical interactions is critical.
• Efficiency: By avoiding explicit 3D reconstruction or external physics engines, the method offers a more flexible and scalable approach to physics-aware video generation.
• Limitations: Current low-level control is limited to velocity; it does not explicitly model friction, mass, or air resistance (though mass awareness was observed implicitly).
• Future Directions: The authors suggest extending the framework to soft-body interactions and incorporating more granular physical parameters (friction, mass) into the conditioning signal to further enhance physical accuracy.

In summary, KineMask bridges the gap between generative video models and physical simulation by teaching VDMs to infer causal rigid body dynamics from minimal initial conditions, achieving state-of-the-art results in both synthetic and real-world scenarios.