🤖 machine learning

Go-with-the-Track: Video Compositing and Motion Control with Point Tracking

Go-with-the-Track is a unified video generation framework that combines point tracking with multiple reference images to enable precise motion control and high-fidelity compositing throughout a video sequence by introducing spatially-aware point-track embeddings and a hybrid training strategy.

Original authors: Koichi Namekata, Yash Kant, Zhizheng Liu, Ryan D Burgert, Yuancheng Xu, Kuan Heng Lin, Emmett Steven, Julien Philip, Li Ma, Andrea Vedaldi, Paul Debevec, Ning Yu

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Koichi Namekata, Yash Kant, Zhizheng Liu, Ryan D Burgert, Yuancheng Xu, Kuan Heng Lin, Emmett Steven, Julien Philip, Li Ma, Andrea Vedaldi, Paul Debevec, Ning Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a film director trying to edit a movie. You have a specific vision: you want a character from a photo to walk into your scene, interact with the environment, and move exactly the way you want them to.

Currently, existing AI video tools are like clumsy assistants. Some can make a character move, but they can only start the character in the very first frame of the video. If you want the character to walk in halfway through the scene, the AI gets confused. Other tools can take a photo and put it into a video, but they have no idea how to make that photo "dance" or move naturally; it just sits there or moves in a generic, robotic way.

"Go-with-the-Track" is a new AI tool that acts like a master puppeteer. It solves both problems at once by using something called "point-tracks."

The Core Idea: The Invisible String

Think of a point-track as an invisible string tied to a specific spot in your video (like the tip of a nose, a wheel on a car, or a leaf on a tree). As the video plays, this string follows that spot frame-by-frame.

In the past, these strings were only tied to the first frame. If you wanted to add a new object later, you couldn't tie a string to it because it wasn't there yet.

Go-with-the-Track changes the rules. It allows you to tie these invisible strings not just to the video, but also to your reference photos.

The Metaphor: Imagine you have a photo of a red ball. You draw a dot on the ball in the photo. Then, you draw a dot on the ball in your video. The AI now knows: "The red ball in the photo is the same as the red ball in the video, and it must follow this specific path."

This lets you tell the AI: "Take the red ball from this photo, and make it roll exactly along this path in the video, even if it enters the scene halfway through."

How It Works (The Magic Tricks)

The paper describes three main "tricks" the AI uses to make this work so well:

1. The "Smart ID Card" (Spatially-Aware Embeddings)
Previously, when AI tried to track thousands of dots, it gave each dot a random, meaningless ID number (like "ID #492"). It was like trying to find a friend in a crowd by shouting random numbers.

The Innovation: Go-with-the-Track gives each dot a "Smart ID Card" based on its actual location and movement. It's like giving your friend a name tag that says "I am the person walking near the tree." Because the ID is based on where the dot actually is, the AI can instantly understand which dot in the photo matches which dot in the video, even if they are far apart. This makes the tracking incredibly precise.

2. The "Compression Suit" (The Adapter)
The AI video model works in a "compressed" world (like a low-resolution sketch) to save energy, but your point-tracks are high-definition details. Usually, squishing high-def details into a low-res sketch loses the fine movements (like a subtle head turn).

The Innovation: The team built a special "adapter" (a compression suit) that fits the high-definition point-tracks perfectly into the AI's low-res world. It keeps all the tiny, important details of the movement without losing any information, ensuring the character doesn't look like a blurry blob.

3. The "Training Gym" (Hybrid Data)
To learn how to move things perfectly, the AI needs to practice. Real-world videos are messy; the "strings" (point-tracks) often break or get lost.

The Innovation: Instead of just practicing on messy real-world videos, the team trained the AI on a mix of:
- Perfect Synthetic Data: Computer-generated worlds where the AI knows exactly where every dot is (like a video game with perfect physics).
- Static Scenes: Rooms where nothing moves, so the AI learns to handle the camera moving.
- Real Videos: To learn how real people and objects look.
  This mix taught the AI to be precise (from the synthetic data) while still looking realistic (from the real data).

What Can You Actually Do With It?

The paper demonstrates several specific applications where this "puppeteer" skill shines:

Video Restylization: You can take a video of a person walking and tell the AI, "Make this person look like a painting from a specific photo." The AI keeps the exact walking motion but changes the look to match the photo.
Mesh-Driven Compositing: You can take a 3D animation (like a robot arm moving) and tell the AI to replace the robot with a specific character from a photo, making the character move exactly like the robot.
Camera Control: You can take a video of a static room or a moving car and tell the AI, "Move the camera to a new angle." The AI uses the point-tracks to understand the 3D space and generates the video from the new angle, keeping the objects in the right place.
Multi-Reference: You can use multiple photos. For example, you can tell the AI to use a photo for the character's face, a different photo for their shirt, and a third photo for the background, and it will stitch them all together while following the motion.

The Bottom Line

Go-with-the-Track is a tool that finally gives filmmakers and creators precise control over both what appears in a video and how it moves. It stops the AI from guessing and starts letting the user direct the action, using simple dots and photos to guide the generation of complex, cinematic videos.

Technical Summary: Go-with-the-Track

Problem Statement

Filmmaking and high-quality video generation require precise control over both the visual appearance of elements (compositing) and their spatiotemporal evolution (motion control). Existing methods typically treat these capabilities in isolation. Point-track-conditioned image-to-video models generally restrict content insertion to the first frame, assuming all controllable elements are present initially; consequently, they struggle to naturally compose elements that enter or exit a scene at arbitrary frames. Conversely, reference-to-video models allow for multi-reference appearance injection but lack fine-grained, editable representations of motion, making it difficult to control how multiple reference elements move and interact across space and time. Furthermore, prior approaches to point-track conditioning often rely on random embeddings to distinguish tracks, which fails to leverage spatial proximity cues, leading to slow convergence and poor generalization when handling large numbers of tracks or discontinuous correspondences between reference images and generated frames.

Methodology

The authors propose Go-with-the-Track, a unified framework that jointly conditions video generation on multiple reference images and reference-anchored point-tracks. The core innovation is the extension of conventional point-tracks to explicitly establish correspondences between generated video frames and input reference images, allowing the model to dictate not only how an element moves but what specific reference content is anchored to that movement.

Key Technical Components

Reference-Anchored Point-Tracks:
Unlike traditional tracks defined solely within a video sequence, the authors define point-tracks that span both the generated video frames and the input reference images. This formulation explicitly links 2D pixel coordinates in the reference images to their trajectories in the generated video, enabling precise compositing of elements that may appear at different times or from different viewpoints.
Spatially-Aware Point-Track Embeddings:
To address the limitation of random embeddings used in prior work, the authors introduce a spatially-aware embedding strategy.
- Encoding: For each point-track, the sequence of 2D coordinates (augmented with frame indices) from the generated frames is processed through a coordinate-wise Multi-Layer Perceptron (MLP) followed by sinusoidal encoding.
- Pooling: A temporal max-pooling operation aggregates these coordinate embeddings into a single vector per track.
- Benefit: This design ensures that the embedding similarity correlates directly with spatial proximity. Tracks that are spatially close in the video share similar embedding characteristics, providing the model with explicit spatial priors to distinguish and associate tracks across frames and reference images, even when the raw coordinates are disjoint.
Lightweight Point-Track Adapter:
A critical challenge is bridging the resolution gap between high-resolution pixel-space point-tracks and the compressed patchified latent space of video diffusion transformers (typically downsampled 16×16 spatially and 4× temporally).
- Block Aggregation: The video volume is divided into $4 \times 16 \times 16$ spatiotemporal blocks, each corresponding to a single latent token.
- Relative Position Injection: For each block, the adapter collects all point-track embeddings falling within it. Crucially, before pooling, each embedding is concatenated with its relative position $(f', x', y')$ within the block.
- Pooling: An MLP processes these concatenated vectors, followed by max-pooling to produce a single compressed conditioning vector per block. This approach preserves fine-grained motion details that naive subsampling or simple downsampling would discard.
Hybrid Training Strategy:
To overcome the scarcity of ground-truth point-track annotations in real-world videos, the authors employ a hybrid training dataset comprising:
- Synthetic Datasets (e.g., PointOdyssey): Provide per-vertex point-tracks derived from 3D meshes, offering perfect ground truth for disentangling camera and object motion.
- Static Scene Datasets (e.g., DL3DV, TartanAir): Provide accurate depth and camera poses to compute precise point-tracks.
- Real Dynamic Video Datasets (e.g., OpenVidHD, MiraData): Provide photorealistic priors, annotated using off-the-shelf trackers (DELTA) with iterative densification to fill sparse regions.
  This mixture allows the model to learn precise motion control from synthetic data while retaining strong photorealistic capabilities from real data.

Key Contributions

Unified Motion Control and Compositing: The paper introduces a paradigm where motion control and multi-reference compositing are unified via reference-anchored point-tracks, enabling spatiotemporally controllable video generation from multiple reference images.
Spatially-Aware Embeddings & Adapter: The authors demonstrate that random embeddings are suboptimal for reference-anchored conditioning. They propose spatially-aware embeddings that encode spatial characteristics and a lightweight adapter that maps pixel-space tracks to latent space without significant motion detail loss.
Hybrid Training Data: By combining synthetic, static, and real-world datasets, the method reduces reliance on noisy point trackers, enabling the model to learn precise motion control while maintaining photorealism.
Broad Applicability: The framework supports diverse applications, including motion-preserved video restylization, mesh- or keypoint-driven compositing, and camera control for both static and dynamic scenes.

Results

Extensive experiments on the DAVIS 2017 and TAPVid3D-ADT datasets demonstrate that Go-with-the-Track outperforms state-of-the-art baselines (including DiffusionAsShader, Go-with-the-Flow, Tora, ATI, and Wan-Move) across visual fidelity, reconstruction accuracy, and motion fidelity metrics.

Quantitative Performance: The model achieves superior scores in FID, FVD, LPIPS, PSNR, SSIM, and Endpoint Error (EPE), particularly in dense track scenarios where baselines often fail or require heavy subsampling.
Ablation Studies: Removing spatially-aware embeddings, the relative position injection in the adapter, or the hybrid training data leads to significant performance degradation, confirming the necessity of each component.
User Study: In a study with 45 participants, Go-with-the-Track was preferred over the second-best method by a margin of more than 2:1 regarding motion following, subject preservation, and overall quality.

Significance

The paper claims that Go-with-the-Track addresses a fundamental gap in video generation by unifying appearance and motion control into a single, flexible framework. By extending point-tracks to serve as spatiotemporal anchors for multiple reference images, the method unlocks new capabilities for filmmakers and creators, such as inserting characters or props at arbitrary frames, restyling videos while preserving complex motion, and retargeting camera movements in both static and dynamic scenes. The work highlights that precise controllability in video generation requires not just better models, but also better conditioning representations that respect the spatial and temporal continuity of the generated content.