Training-free Motion Factorization for Compositional Video Generation

This paper proposes a training-free, model-agnostic motion factorization framework that decomposes complex video generation into motionlessness, rigid, and non-rigid categories through a "planning before generation" paradigm to synthesize diverse instances with controlled appearance and motion.

The Gist

Imagine you are a movie director trying to film a complex scene with a computer. You type a prompt like: "A car drives past a waving flag, while an ancient building stands in the background."

In the past, AI video generators were like enthusiastic but confused interns. They heard "car," "flag," and "building," but they didn't quite understand how each one should move.

• They might make the building wobble like jelly.
• They might make the flag stiff as a board.
• Or they might make the car float like a ghost.

This paper introduces a new system called Motion Factorization. Think of it as hiring a smart choreographer and a specialized film crew to fix the mess. The system doesn't need to be retrained (it's "training-free"); it just uses a clever set of rules to organize the chaos before the video is even made.

Here is how it works, broken down into simple steps:

1. The "Motion Graph" (The Choreographer's Script)

Before the AI starts drawing frames, it first reads your prompt and builds a Motion Graph. Imagine this as a flowchart or a script for a play.

• The Problem: The word "drives" is vague. Does the car spin? Does it shake?
• The Solution: The system uses a Large Language Model (like a super-smart robot brain) to break your sentence down into a structured map.
  • The Building: It sees "stands" and labels this as "Motionless." (Like a statue).
  • The Car: It sees "drives" and labels this as "Rigid Motion." (Like a solid box sliding across the floor).
  • The Flag: It sees "waving" and labels this as "Non-Rigid Motion." (Like a piece of cloth flapping in the wind).

This step solves the confusion. The AI now knows exactly what kind of movement each object needs before it draws a single pixel.

2. The "Disentangled Guidance" (The Specialized Crew)

Once the script is written, the system sends the instructions to three different "specialist crews" to handle the actual video generation. Instead of giving everyone the same instructions, it tailors them:

• Crew A: The Anchors (For Motionless Objects)

  • Task: Keep the building perfectly still.
  • Analogy: Imagine the building is a painting on a wall. This crew makes sure the painting doesn't flicker, shake, or change color from frame to frame. They "anchor" the image so it looks stable.

• Crew B: The Rigid Sliders (For Moving Objects)

  • Task: Move the car.
  • Analogy: Imagine the car is a solid Lego block. This crew slides the block across the screen. They make sure the Lego block doesn't stretch, squash, or turn into a blob. It stays a perfect car shape, just in a different spot.

• Crew C: The Stretchy Artists (For Waving Objects)

  • Task: Make the flag wave.
  • Analogy: Imagine the flag is made of wet silk. This crew allows the pixels to wiggle, stretch, and twist. They don't force the flag to stay rigid; they let it flow naturally like fabric in the wind.

3. The Result

By separating the instructions this way, the final video looks much more realistic.

• The building stays rock solid.
• The car drives smoothly without turning into a melting puddle.
• The flag flutters realistically.

Why is this a big deal?

Most AI video tools try to learn everything at once, which often leads to "motion blur" where everything looks like it's moving the same way. This paper says, "Stop trying to learn everything at once. Just categorize the movement first, then apply the right rule to each object."

It's like the difference between a chaotic dance party where everyone is bumping into each other, and a well-rehearsed ballet where the dancers know exactly when to stand still, when to slide, and when to spin.

In short: This paper gives AI a "traffic cop" for video generation, directing different objects to follow different rules of physics, resulting in videos that actually make sense.

Technical Summary

Here is a detailed technical summary of the paper "Training-free Motion Factorization for Compositional Video Generation."

1. Problem Statement

Compositional Video Generation (CVG) aims to synthesize videos containing multiple interacting instances with diverse appearances and motion types based on complex user prompts. However, current state-of-the-art models face two critical limitations:

• Motion Homogenization: Existing models tend to generate similar motion patterns for all objects, failing to distinguish between distinct motion categories (e.g., a static building vs. a waving flag vs. a driving car).
• Semantic Ambiguity: Directly translating natural language prompts into motion trajectories (e.g., bounding box sequences) often leads to broken paths, abnormal size variations, and implausible dynamics due to linguistic ambiguity.
• Rough Guidance: Standard diffusion guidance mechanisms apply uniform constraints across the scene, failing to differentiate between static regions, rigid bodies, and non-rigid deformations.

2. Methodology

The authors propose a training-free framework that decomposes complex scene dynamics into three canonical motion categories: Motionlessness, Rigid Motion, and Non-rigid Motion. The framework follows a "Planning before Generation" paradigm, consisting of two model-agnostic modules:

A. Structured Motion Reasoning (SMR)

This module acts as a planner to resolve semantic ambiguities before video generation begins.

• Motion Graph Construction: Instead of inferring motion directly from text, the system uses a Large Language Model (LLM) to parse the user prompt into a Motion Graph.
  • Nodes: Represent instances with attributes and a canonical motion label (static, rigid, or non-rigid).
  • Edges: Encode spatial relationships (e.g., "next to") and dynamic interactions (e.g., "pass by").
• Layout Reasoning: Based on the graph, the LLM infers frame-wise bounding box sequences ($B_1, \dots, B_F$) for each instance:
  • Motionlessness: Position and size remain constant across frames.
  • Rigid Motion: Position updates via global velocity and acceleration vectors; size remains constant.
  • Non-rigid Motion: Modeled by multiple local motion vectors allowing for asymmetric shifts and size changes (deformation).

B. Disentangled Motion Guidance (DMG)

This module guides the video diffusion model during the synthesis process using specialized branches for each motion category. It operates without retraining the base model.

• Reference Conditioned Guidance (for Motionlessness): To prevent flickering in static regions, the system identifies a stable "reference frame" (minimum feature difference) and anchors pixel-wise features of static instances to this frame across all time steps.
• Geometric Invariance Guidance (for Rigid Motion): To preserve the shape of moving rigid objects (e.g., cars), the system generates a frame-agnostic shape template via k-means clustering and pixelwise consensus. It restricts cross-frame interactions to geometrically aligned regions, preventing shape distortion.
• Spatial Deformation Guidance (for Non-rigid Motion): To handle complex deformations (e.g., waving flags, dancing), the system minimizes the discrepancy between perceptual deformations (derived from diffusion feature nearest-neighbor search) and box-induced deformations (derived from bounding box corner displacements). This ensures the generated deformation aligns with the intended motion vector.

Integration: The guidance signals are integrated into the attention mechanism of the diffusion model (either 3D U-Net or DiT architectures) by modulating attention maps with motion-specific masks ($G_m, G_r, G_{nr}$).

3. Key Contributions

1. Motion Factorization: A novel decomposition of scene dynamics into three distinct categories (static, rigid, non-rigid), enabling targeted modeling of diverse motion behaviors.
2. Structured Motion Graph: The introduction of a motion graph as an intermediate representation to bridge the gap between ambiguous natural language prompts and precise spatial-temporal layouts.
3. Disentangled Guidance Mechanism: Specialized guidance branches that enforce appearance consistency, geometric invariance, and local deformation control, respectively, without requiring model fine-tuning.
4. Model Agnosticism: The framework is designed to be seamlessly integrated into various diffusion architectures (demonstrated on both 3D U-Net and DiT models).

4. Experimental Results

The authors constructed new benchmarks (CVGBench-m from MSR-VTT and CVGBench-p from Panda-70M) featuring diverse compositional prompts.

• Quantitative Performance:
  • The framework was tested on VideoCrafter-v2.0 (3D U-Net) and CogVideoX-2B (DiT).
  • It achieved state-of-the-art results across five metrics: Subject Consistency, Background Consistency, Temporal Flickering, Motion Smoothness, and Dynamic Degree.
  • Example: On VideoCrafter-v2.0, the framework improved Subject Consistency from ~97.5% (baseline) to 98.40% and Dynamic Degree from ~38% to 82.21%.
• Qualitative Analysis:
  • Static: Successfully suppressed unwanted movement in static scenes (e.g., buildings, standing people).
  • Rigid: Preserved object integrity and geometry during translation (e.g., driving cars) without warping.
  • Non-rigid: Generated expressive and coherent deformations for dynamic objects (e.g., dancing, fighting) where baselines often failed to maintain pose progression.
• Ablation Studies:
  • Removing the SMR module caused significant drops in consistency and smoothness, proving the necessity of structured reasoning.
  • Larger LLM backbones (70B vs. 8B) yielded better reasoning quality.
  • All three guidance branches (RCG, GIG, SDG) contributed incrementally to performance improvements.

5. Significance

This paper addresses a fundamental bottleneck in video generation: the inability to control how different objects move within a single scene. By factorizing motion and providing a training-free, plug-and-play solution, the framework significantly enhances the realism and controllability of compositional video generation. It demonstrates that explicit reasoning about motion laws (via graphs) and disentangled guidance can outperform end-to-end learning approaches that rely solely on prompt conditioning. This approach is particularly valuable for applications requiring precise control over complex interactions, such as virtual reality, simulation, and storytelling.