💻 computer science

TrackMAE: Video Representation Learning via Track Mask and Predict

TrackMAE is a self-supervised video representation learning method that improves upon traditional masked video modeling by explicitly leveraging motion trajectories from point trackers to guide motion-aware masking and provide complementary motion targets for reconstruction, resulting in superior performance on motion-centric tasks.

Original authors: Renaud Vandeghen, Fida Mohammad Thoker, Marc Van Droogenbroeck, Bernard Ghanem

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Renaud Vandeghen, Fida Mohammad Thoker, Marc Van Droogenbroeck, Bernard Ghanem

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand a movie.

The Old Way: The "Blurry Snapshot" Problem
Previously, researchers taught robots using a method called "Masked Video Modeling." Think of this like playing a game of "Mad Libs" with a movie. You take a video, cover up 90% of the frames with black boxes, and ask the robot: "Based on the few frames you can see, what do you think the hidden parts look like?"

The robot usually gets good at guessing the colors and textures. If it sees a red car in one frame, it guesses the hidden frames also have a red car. But here's the problem: the robot often ignores movement. It learns that a red car looks like a red car, but it doesn't really understand that the car is driving from left to right. It's like learning to recognize a person by their face, but not understanding how they walk or wave.

Because of this, these robots are great at identifying what is in a video (e.g., "That's a dog"), but terrible at understanding actions that rely on motion (e.g., "The dog is jumping over a fence" vs. "The dog is sitting").

The New Solution: TrackMAE (The "Sticky Note" Method)
The authors of this paper, TrackMAE, decided to fix this by giving the robot a new tool: Motion Tracking.

Imagine you are watching a video of a person juggling.

The Tracker: Before the robot tries to guess the hidden parts, the system uses a super-smart "point tracker" (like a digital sticky note) to follow specific dots on the objects. It watches a dot on the ball, a dot on the juggler's hand, and a dot on the floor.
The Trajectory: The system draws a line showing exactly where those dots moved from frame 1 to frame 10. This is the "motion trajectory."

How the Training Game Changes
Now, the training game has two parts:

The Visual Guess: "What did the hidden pixels look like?" (The old way).
The Motion Guess: "Based on the visible frames, where did the sticky notes move to in the hidden frames?"

The robot now has to learn two things at once: What does the object look like? AND How did it move?

The "Motion-Aware" Masking Trick
The authors also changed how they hide the video.

Old Way: They hid random parts of the screen. Sometimes they hid a static wall; sometimes they hid a moving car. It was a coin flip.
New Way (Motion-Aware): They look at the motion tracker first. They make sure to hide a mix of moving things (like the juggling balls) and still things (like the background wall).
- Why? If you only hide the background, the robot gets lazy. If you only hide the moving parts, it gets confused. By forcing the robot to solve puzzles for both moving and still objects, it learns a much deeper understanding of the scene.

The Result: A Smarter Robot
Because the robot is forced to predict where things go, not just what they look like, it becomes much better at:

Fine-grained actions: Distinguishing between "throwing a ball" and "catching a ball."
Generalization: If it learns to understand motion in a video of a gymnast, it can easily understand motion in a video of a dancer, even if it's never seen a dancer before.

In a Nutshell
Think of the old models as students who memorized the script of a play but forgot the choreography.
TrackMAE is a student who memorized the script and practiced the dance moves. When the lights go out (the video is masked), this student can still tell you exactly what happened because they understand the flow of the action, not just the static images.

This makes the AI much better at tasks that require understanding time, speed, and direction, which are crucial for real-world video understanding.

1. Problem Statement

Masked Video Modeling (MVM) has become a dominant paradigm for self-supervised video pretraining. However, standard MVM approaches (e.g., VideoMAE) suffer from two primary limitations:

Implicit Motion Encoding: They rely on pixel reconstruction with random tube masking. This often allows models to solve the task using low-level spatial correlations (color/texture continuity) or short-horizon consistency without learning fine-grained temporal dynamics.
Poor Performance on Motion-Centric Tasks: Consequently, these models perform well on appearance-heavy datasets (e.g., Kinetics-400) but struggle on tasks requiring precise temporal understanding (e.g., Something-Something V2, FineGym), where the "how" of an action is more critical than the "what."

Existing attempts to inject motion structure either bias masking toward high-motion regions (using optical flow) or replace pixel targets with feature targets (e.g., DINO, CLIP). However, these methods still treat motion as an implicit signal rather than an explicit reconstruction target, limiting the model's ability to learn identity preservation and trajectory dynamics.

2. Methodology: TrackMAE

The authors propose TrackMAE, a novel MVM framework that explicitly incorporates motion information as a reconstruction signal. The method consists of three core components:

A. Motion Target Extraction (CoTracker3)

Instead of relying on pre-computed optical flow (which is noisy and computationally heavy for long-range tracking), TrackMAE uses an off-the-shelf point tracker, CoTracker3, to generate motion trajectories.

Process: Query points are sampled from a uniform grid on the first frame of a video clip. CoTracker3 tracks these points across subsequent frames.
Target Definition: The model predicts the displacement of these points rather than absolute coordinates.
Upsampling Trick: To avoid the high computational cost of dense tracking, the model tracks only one point per patch (sparse tracking). These sparse motion tokens are then spatially upsampled (via interpolation) to match the density of the video tokens. This allows the model to learn dense motion semantics without the computational overhead of dense tracking.

B. Dual-Decoder Architecture

TrackMAE employs a standard ViT encoder but utilizes two separate decoders to handle distinct reconstruction tasks simultaneously:

Spatial Decoder ( $\Psi_{spatial}$ ): Reconstructs the input in either pixel space or a semantic feature space (e.g., CLIP features).
Motion Decoder ( $\Psi_{motion}$ ): A lightweight decoder that predicts the extracted motion trajectories (displacements) for the masked tokens.

The total loss function is a weighted sum:
$\mathcal{L} = \mathcal{L}_{spatial} + \lambda \cdot \mathcal{L}_{motion}$
Where $\mathcal{L}_{motion}$ is an L2 loss on the predicted vs. ground-truth motion displacements for masked tokens only.

C. Motion-Aware Masking

Standard random tube masking ignores motion content. TrackMAE introduces a motion-aware masking strategy:

Mechanism: Using the full trajectory predictions from CoTracker3, the system computes the average displacement magnitude for each patch.
Sampling: Visible tokens are sampled from two bins: high-motion and low-motion regions.
Balance: The strategy ensures that the visible context contains a balanced mix of static and dynamic regions (controlled by a ratio $\rho_{motion}$ ), forcing the model to infer motion dynamics from limited visible cues rather than relying on static spatial correlations.

3. Key Contributions

Explicit Motion Supervision: TrackMAE is the first MVM approach to use tracked point trajectories as a direct reconstruction target, complementing spatial (pixel/feature) reconstruction.
Motion-Aware Masking: A novel masking strategy that ensures the visible context covers both high- and low-motion regions, preventing the model from ignoring temporal dynamics.
Efficient Dense Motion Learning: The introduction of a spatial upsampling trick allows the model to learn from dense motion targets using only sparse tracking, significantly reducing computational costs.
State-of-the-Art Performance: The method achieves superior results on motion-centric benchmarks while maintaining strong performance on appearance-based tasks.

4. Experimental Results

The authors evaluated TrackMAE on six datasets across linear probing, full fine-tuning, and generalization benchmarks (SEVERE).

Linear Probing:
- On motion-centric datasets (Something-Something V2, FineGym), TrackMAE significantly outperforms previous SOTA methods (e.g., VideoMAE, MGMAE, MGM). For instance, on SSv2, it improves over the best pixel-based baseline by ~2.6%.
- On appearance-centric datasets (Kinetics-400), it performs on par with or slightly better than feature-based methods (e.g., SMILE, SIGMA).
Full Fine-Tuning:
- TrackMAE with CLIP targets achieves 72.8% Top-1 on SSv2 and 83.9% on Kinetics-400, surpassing all prior masked video modeling methods and competing with strong supervised baselines.
Generalization (SEVERE Benchmark):
- TrackMAE demonstrates robustness across domain shifts, sample efficiency (low-data settings), action granularity, and task shifts. It consistently outperforms baselines in fine-grained action recognition and low-shot scenarios.
Ablation Studies:
- Trajectory + Spatial: Combining motion targets with spatial targets yields the best results, confirming they are complementary.
- Masking: Motion-aware masking provides consistent gains (~0.5%) over random masking.
- Upsampling: Upsampling sparse tracks (14x14 $\to$ 28x28) improves performance without added computational cost compared to native dense tracking.

5. Significance and Impact

Bridging the Gap: TrackMAE successfully bridges the gap between appearance-based and motion-based representation learning. It proves that explicit motion supervision is crucial for learning discriminative video representations, especially for tasks where temporal dynamics define the action.
Efficiency: By leveraging sparse tracking with upsampling, the method offers a computationally efficient way to inject high-quality motion signals into MVM, avoiding the heavy preprocessing costs of optical flow or dense tracking.
Generalization: The results suggest that learning to predict "how things move" (trajectories) alongside "what things look like" (pixels/features) creates a more robust and generalizable video foundation model, capable of handling distribution shifts and fine-grained distinctions better than previous SSL methods.

In conclusion, TrackMAE redefines the masked video modeling paradigm by treating motion not just as a byproduct of video data, but as a primary signal to be explicitly predicted and reconstructed, leading to state-of-the-art performance across diverse video understanding tasks.