💻 computer science

MotionGrounder: Grounded Multi-Object Motion Transfer via Diffusion Transformer

This paper introduces MotionGrounder, a Diffusion Transformer-based framework that enables controllable multi-object motion transfer in video generation by utilizing a Flow-based Motion Signal for stable motion priors and an Object-Caption Alignment Loss to ground object captions to their spatial regions, outperforming existing single-object methods across various evaluations.

Original authors: Samuel Teodoro, Yun Chen, Agus Gunawan, Soo Ye Kim, Jihyong Oh, Munchurl Kim

Published 2026-04-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Samuel Teodoro, Yun Chen, Agus Gunawan, Soo Ye Kim, Jihyong Oh, Munchurl Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a home video of your dog chasing a ball in the park. Now, imagine you want to make a new video where a robot dog is chasing a glowing ball on a neon-lit city street, but you want the robot dog to move exactly the same way your real dog did in the original video.

This is the magic of MotionGrounder.

The Problem: The "Confused Director"

Before this paper, AI video tools were like directors who could only handle one actor at a time.

If you asked for a video of a "wolf and a bear," the AI would get confused. It might make the wolf walk like the bear, or it might make the bear disappear entirely.
Existing tools were great at copying movement (the "dance"), but they were terrible at knowing who was dancing. They would mix up the actors, leading to chaotic scenes where objects swapped places or moved in the wrong direction.

The Solution: MotionGrounder

The researchers at KAIST built a new system called MotionGrounder. Think of it as a super-smart film director who has two special tools to keep the scene organized:

1. The "Stable Dance Instructor" (Flow-based Motion Signal)

Imagine trying to teach someone a dance by just watching a blurry, shaky video. They would probably trip over their own feet.

Old AI: Tried to copy the dance by looking at the blurry pixels, which often led to "noisy" or shaky movements.
MotionGrounder: Uses a "Flow-based Motion Signal." Think of this as a laser-guided dance instructor. Instead of looking at the blurry pixels, it calculates the exact path every object takes (like drawing a smooth line under the dancer's feet). This ensures the new robot dog moves smoothly and naturally, just like the real dog, without getting jittery.

2. The "Name Tag System" (Object-Caption Alignment)

This is the real game-changer. Imagine a classroom where the teacher says, "The student in the red shirt, stand up!"

Old AI: Might make everyone stand up, or make the student in the blue shirt stand up because it got confused. It didn't know which object belonged to which word.
MotionGrounder: Uses Object-Caption Alignment. It puts invisible name tags on every object in the video.
- It sees the "Wolf" in the video and attaches a tag that says "Wolf."
- It sees the "Bear" and attaches a tag that says "Bear."
- When you type a new caption like "A wolf and a bear," the AI looks at the tags. It knows, "Okay, the Wolf tag goes to the Wolf, and the Bear tag goes to the Bear."
- This ensures the wolf walks like the wolf, and the bear walks like the bear. They don't get mixed up.

How It Works in Real Life

The paper shows some cool examples:

Scenario A: A soldier walks toward a tank in a desert.
- MotionGrounder can take that walking motion and apply it to a robot dog walking toward a hover drone in a neon city. The robot dog walks exactly like the soldier did, but it stays in its own spot and doesn't turn into the soldier.
Scenario B: A fox and a raccoon are sitting on a log.
- MotionGrounder can change them into a child and a rabbit sitting on a fallen tree. The child moves like the fox, and the rabbit moves like the raccoon, perfectly keeping their identities separate.

Why Is This a Big Deal?

No Training Needed: You don't need to teach the AI new tricks. It works "out of the box" (Zero-Shot). You just give it a video and a description, and it figures it out.
Multi-Object Mastery: It's the first tool that can handle complex scenes with many moving parts without getting confused.
Precision: It doesn't just guess; it mathematically ensures that the "Wolf" in your text matches the "Wolf" in the video.

The Catch (Limitations)

Like any new technology, it has limits. If you try to put too many objects in the scene (like 6 or 8 animals dancing at once), the AI starts to get a bit overwhelmed, just like a human director trying to manage a massive crowd. But for most normal scenes (1 to 3 or 4 objects), it works beautifully.

In a Nutshell

MotionGrounder is like giving an AI a pair of glasses that let it see exactly who is doing what. It takes the "dance moves" from one video and teaches them to new characters in a new setting, making sure the right character does the right move, every single time.

1. Problem Statement

Existing text-to-video (T2V) generation methods, particularly those based on Diffusion Transformers (DiTs), have made significant strides in visual fidelity. However, motion transfer—the process of transferring temporal dynamics from a reference video to a new video conditioned on a text caption—faces critical limitations in multi-object scenarios:

Single-Object Restriction: Most current methods are limited to single-object motion transfer. They struggle to handle real-world scenes containing multiple interacting objects.
Lack of Semantic Grounding: Even methods attempting multi-object transfer (e.g., ConMo, MultiMotion) often lack explicit association between specific object captions and their spatial regions. This leads to motion misattribution (e.g., assigning the motion of a wolf to a bear) and spatial misalignment (generating objects in the wrong locations).
Noisy Motion Signals: DiT-based approaches that rely on attention maps (like DiTFlow) often extract noisy motion signals, hindering precise control.
Dependency on Inversion: Some high-quality methods rely on costly DDIM inversion, which is incompatible with efficient DiT architectures.

The core challenge is to achieve training-free, zero-shot motion transfer that simultaneously preserves complex motion dynamics and ensures explicit, grounded alignment between multiple object captions and their corresponding spatial regions in the generated video.

2. Methodology

MotionGrounder is a training-free, DiT-based framework designed to solve the multi-object motion transfer problem. It operates by injecting two novel components into the denoising process of a pre-trained DiT (specifically CogVideoX-5B in the experiments):

A. Flow-based Motion Signal (FMS)

To replace noisy attention-based motion signals with a stable prior, the authors propose FMS:

Optical Flow Estimation: Instead of relying on internal attention maps, FMS estimates dense optical flow from the source video frames using an external model (GMFlow).
Latent Trajectory Construction: The optical flow fields are downsampled to the latent space resolution. Patch trajectories are constructed by propagating latent patch locations across frames based on these displacement fields.
Motion Supervision: During the denoising steps, the model computes cross-frame attention displacements. A loss function ( $L_{FMS}$ ) minimizes the difference between these generated displacements and the stable trajectories derived from the optical flow. This ensures the generated video adheres to the source motion dynamics without requiring DDIM inversion.

B. Object-Caption Alignment Loss (OCAL)

To enforce that specific objects appear in specific locations as described in the text, OCAL manipulates the attention mechanisms of the DiT:

Attention Extraction: The model extracts 3D attention maps (both text-to-video and video-to-text) corresponding to specific object tokens in the caption.
Spatial Grounding: These attention maps are aligned with the provided object masks (downsampled to latent resolution).
Loss Formulation: OCAL consists of two terms:
1. Foreground Loss: Maximizes attention values within the region defined by the object mask.
2. Background Loss: Suppresses attention values outside the object mask.
Size Regularization: A weight is applied based on object size to prevent large objects from dominating the optimization, ensuring smaller objects are also generated correctly.

C. Overall Objective

The total optimization objective at each timestep combines the motion supervision and the grounding alignment:
$L_t = \lambda_{FMS} \cdot L_{FMS} + \lambda_{OCAL} \cdot L_{OCAL}$
This is optimized via a few steps of gradient descent during the early phase of the denoising process (training-free inference).

3. Key Contributions

First Multi-Object Grounded Transfer: MotionGrounder is the first DiT-based framework to handle joint motion transfer and semantic grounding for scenes with multiple objects without additional training.
Stable Motion Prior (FMS): Introduces a flow-based motion signal that provides stable, object-consistent motion guidance, overcoming the noise issues of attention-flow methods.
Explicit Grounding Mechanism (OCAL): Proposes a novel loss function that explicitly enforces the spatial alignment between object captions and their designated regions in the video, solving the "motion misattribution" problem.
Object Grounding Score (OGS): Introduces a new evaluation metric that jointly assesses:
- Spatial Alignment: Intersection over Union (IoU) between source and generated object regions.
- Semantic Consistency: Local CLIP similarity between the generated object crop and its specific caption.
- This metric addresses the failure of global CLIP scores to detect object-level swapping or misplacement.

4. Experimental Results

The authors evaluated MotionGrounder on a custom dataset of 52 multi-object videos (from DAVIS, YouTube-VOS, and ConMo) across three prompt settings: Caption (same content), Subject (change objects, keep background), and Scene (change everything).

Quantitative Performance: MotionGrounder outperformed five state-of-the-art baselines (DMT, MOFT, MotionClone, ConMo, DiTFlow) across almost all metrics.
- It achieved the highest Object Grounding Score (OGS), IoU, and Local Textual Alignment (LTA).
- It maintained high Motion Fidelity (MF), comparable to or better than baselines, proving it does not sacrifice motion quality for grounding.
Qualitative Results: Visual comparisons showed that baselines often failed to generate all objects, swapped object motions (e.g., a soldier moving like a helicopter), or placed objects in incorrect locations. MotionGrounder successfully generated all specified objects with correct motion and spatial placement.
User Study: In a study with 49 participants, MotionGrounder received the highest rankings for Motion Adherence, Global Textual Faithfulness, and Object Grounding.
Ablation Studies:
- Removing FMS led to poor motion preservation.
- Removing OCAL led to object entanglement and misplacement.
- The combination of both yielded the best trade-off.
- The method was also validated on the Wan2.1 backbone, demonstrating model-agnostic generalization.

5. Significance and Impact

Advancing Controllable Video Generation: MotionGrounder bridges the gap between high-fidelity video generation and fine-grained control, enabling users to specify complex multi-object interactions with precise motion and spatial constraints.
Practical Applications: This capability is crucial for creative industries (film, advertising), education, simulation, and assistive content creation where specific object behaviors must be controlled.
New Evaluation Standard: The introduction of OGS provides a necessary tool for the community to evaluate object-level grounding, moving beyond global text-video alignment metrics that miss fine-grained errors.
Limitations: The method currently struggles with scenes containing more than 3-4 objects due to the low-resolution nature of latent spaces in DiTs, which causes small objects to be lost or poorly represented. Future work may need to explore higher-resolution latent spaces or pixel-domain processing.

In summary, MotionGrounder represents a significant step forward in zero-shot, training-free video editing, successfully unifying stable motion transfer with rigorous semantic grounding for complex, multi-object scenes.