💻 computer science

TRACE: Object Motion Editing in Videos with First-Frame Trajectory Guidance

The paper presents Trace, a two-stage framework that enables practical and controllable object motion editing in videos by allowing users to design a trajectory in a single anchor frame, which is then mapped to frame-aligned box trajectories and used to regenerate the object while preserving the original scene content.

Original authors: Quynh Phung, Long Mai, Cusuh Ham, Feng Liu, Jia-Bin Huang, Aniruddha Mahapatra

Published 2026-03-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Quynh Phung, Long Mai, Cusuh Ham, Feng Liu, Jia-Bin Huang, Aniruddha Mahapatra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a home video of your child running through a park with a kite. The scene is beautiful, but you wish the kite wasn't flying in a straight line; you want it to swoop in a giant "S" shape or loop around a tree.

In the past, editing a video to change how an object moves was like trying to rewrite a novel by hand-editing every single letter of every single page. You'd have to manually tell the computer exactly where the kite should be in frame 1, frame 2, frame 3, and so on. If the camera was moving, this became a nightmare because the kite's position relative to the background was constantly shifting.

TRACE is a new tool that makes this magic much easier. Think of it as a "Motion Director" for your videos. Here is how it works, broken down into simple concepts:

1. The "First-Frame Map" (The User's Job)

Instead of dragging the kite through 100 different frames, you only need to draw a path on the very first frame of the video.

The Analogy: Imagine you are looking at a map of a city. You draw a line from "Home" to "The Park." You don't need to tell the GPS every single turn it should take at every intersection; you just draw the destination route.
In TRACE: You draw a line (or a few boxes) on the first frame showing where you want the object to go. You are saying, "If the camera stayed still, I want the kite to move here."

2. The "Smart Translator" (The First Stage)

This is the tricky part. In your video, the camera is moving (maybe the person filming is walking). If you just took your "First-Frame Map" and pasted it onto the rest of the video, the kite would look like it was floating in mid-air while the world moved around it.

The Analogy: Imagine you are on a moving train looking out the window. You draw a line on the glass showing where a bird is flying. If the train speeds up or turns, that line on the glass needs to shift to match the bird's new position relative to the passing trees.
In TRACE: The system has a "Smart Translator" module. It looks at your simple drawing on the first frame and the movement of the camera. It calculates, "Okay, the camera is moving left, so to keep the kite on that path, I need to move the kite's position in the next frame slightly to the right." It turns your simple 2D drawing into a complex, 3D-aware path that fits perfectly into the moving video.

3. The "Magic Eraser and Painter" (The Second Stage)

Once the system knows exactly where the object should be in every single frame, it has to actually change the video.

The Analogy: Imagine a painter who is incredibly skilled at two things: erasing a specific person from a photo without smudging the background, and then painting a new version of that person in a different pose, making sure the lighting and shadows match perfectly.
In TRACE: The system "erases" the original object from the video and "paints" it back in, following the new path you designed. It is careful to keep the background (the trees, the sky, the other people) exactly the same, so the video still looks real.

Why is this a big deal?

No 3D Modeling Required: You don't need to be a 3D artist or understand complex geometry. You just draw a line.
It Handles Camera Movement: Most tools break if the camera moves. TRACE understands that the camera is moving and adjusts the object's path accordingly.
It Keeps the "Vibe": It doesn't just slap a new object in; it respects the original lighting, style, and background of your video.

Real-World Examples

The Kite: Change a kite's flight path from a straight line to a figure-eight.
The Deer: Replace a deer walking in a garden with a tiger, but make the tiger walk the exact same path the deer was taking.
The Cyclist: Make a cyclist ride in a circle instead of a straight line, even if the camera is panning to follow them.

In short: TRACE lets you be the director. You draw the path on the first frame, and the AI handles the heavy lifting of calculating the physics, adjusting for camera movement, and re-rendering the video so the object follows your new script perfectly.

1. Problem Definition

The paper addresses the challenge of object-centric motion editing in videos. The goal is to alter the trajectory of a specific target object (e.g., changing a kite's flight path or a person's running direction) while strictly preserving the original scene content, camera motion, background, and the object's appearance.

Key Challenges:

User Input Difficulty: Existing methods often require users to provide dense, per-frame bounding boxes or point tracks, which is tedious and difficult, especially in videos with dynamic camera movements.
Cross-View Misalignment: A user-defined path on a static first frame does not naturally align with the object's position in subsequent frames due to camera motion (parallax, perspective shifts).
Content Preservation: Standard video generation models often hallucinate new backgrounds or alter the object's identity when forced to follow new motion constraints.

2. Methodology: The TRACE Framework

TRACE proposes a novel problem setting: Video Editing with First-Frame Object Motion Design. It allows users to define a desired 2D trajectory on a single anchor frame (the first frame), and the system synthesizes a temporally consistent video where the object follows this path.

The framework operates via a two-stage pipeline:

Stage 1: Cross-View Motion Transformation Module

This module bridges the gap between the user's static first-frame design and the dynamic video view.

Input: A user-defined path (sparse bounding boxes) on the first frame, the first-frame image, and point tracks capturing camera motion.
Mechanism: Instead of relying on error-prone explicit 3D reconstruction (which suffers from noisy depth/pose estimation), the authors use a Diffusion Transformer (DiT) to learn a sequence-to-sequence mapping.
Training: The model is trained on synthetic data (generated using ReCamMaster) where static camera videos are re-rendered with dynamic camera paths. It learns to implicitly model camera dynamics to predict the corresponding per-frame bounding box sequence ( $B_{tgt}$ ) in the video view.
Output: A temporally consistent, frame-aligned bounding box sequence that guides the object to the intended destination despite camera movement.

Stage 2: Video Re-Synthesis Module

This module generates the final video based on the predicted trajectory.

Architecture: Based on the Wan 2.1 video diffusion model (fine-tuned with LoRA).
Conditioning: The model is conditioned on:
1. The original video (masked to remove the object).
2. The first frame (as an appearance reference).
3. The predicted bounding box sequence ( $B_{tgt}$ ) from Stage 1.
4. Text prompts and binary masks (inpainting regions and synthesis regions).
Process: The model performs simultaneous object removal (inpainting) and object generation (synthesis). It "erases" the original object and regenerates it along the new path while ensuring the background and other scene elements remain unchanged.
Data Strategy: To prevent overfitting to specific box guidance, the training data includes smoothed boxes and added noise, and input conditions are randomly dropped.

3. Key Contributions

Novel Problem Setting: Introduced "First-Frame Motion Design," enabling intuitive control over object trajectories via a single anchor frame rather than tedious per-frame annotation.
TRACE Framework: Developed a two-stage pipeline that decouples the complex task of cross-view motion transformation from video synthesis.
- A learnable Cross-View Motion Transformation module that handles camera dynamics without explicit 3D reconstruction.
- A Motion-Conditioned Video Re-Synthesis module that preserves object identity and scene consistency.
Synthetic Data Pipeline: Created a scalable data generation pipeline using ReCamMaster to produce paired static/dynamic videos for training the transformation module, overcoming the lack of real-world paired data.
Versatile Applications: Demonstrated capabilities beyond simple motion editing, including multi-object editing, object insertion, object attribute editing, and object replacement.

4. Experimental Results

The authors evaluated TRACE on real-world videos (DAVIS benchmark and a custom test set) against state-of-the-art Image-to-Video (I2V) and Video-to-Video (V2V) baselines (e.g., MagicMotion, VACE, HunyuanCustom, GenCompositor).

Cross-View Transformation:
- TRACE significantly outperformed baselines (Interpolation, MegaSAM, Depth-Anything) in predicting accurate box trajectories.
- Metrics: Achieved 0.80 IoU and 0.91 mAP (f2v), compared to ~0.53–0.65 mAP for 3D warping baselines. Qualitative results showed TRACE produced smooth, stable paths while baselines resulted in "jittery" or off-track trajectories.
Video Re-Synthesis:
- Quality: Achieved state-of-the-art scores in FVD, FID, and KID (generative quality) and high SSIM/PSNR (similarity to input).
- Alignment: Achieved competitive Tube IoU and mAP, demonstrating precise adherence to the motion path.
- Qualitative: TRACE successfully preserved object identity and background consistency, whereas I2V baselines often hallucinated backgrounds or altered object appearance, and V2V baselines struggled with identity loss.
User Study:
- In a pairwise comparison on 70 videos, users preferred TRACE over all baselines.
- Preference Rates: >77% for video quality (peaking at 85% vs. PISCO) and ~75% for object identity preservation.

5. Significance and Impact

Usability: TRACE democratizes complex video editing by removing the need for technical, frame-by-frame control, making professional-grade motion re-planning accessible to non-experts.
Creative Workflow: It enables new post-production workflows, such as changing the pacing or direction of a specific actor in a scene without re-shooting or manually rotoscoping every frame.
Technical Advancement: The paper demonstrates that learning implicit camera dynamics via a diffusion-based transformer is superior to explicit 3D geometric warping for motion editing tasks, offering a robust solution to the "cross-view misalignment" problem.

In summary, TRACE represents a significant leap forward in controllable video generation, successfully combining intuitive user interaction with high-fidelity, temporally consistent video synthesis.