Denoise to Track: Harnessing Video Diffusion Priors for Robust Correspondence

The paper introduces HeFT, a zero-shot point tracking framework that leverages pretrained video diffusion models by analyzing their internal representations to identify specialized attention heads and low-frequency components, which are then selectively harnessed through a denoising-based strategy to achieve state-of-the-art tracking performance without annotated training data.

The Gist

Imagine you are trying to follow a specific red balloon as it floats through a crowded, chaotic parade. The balloon gets hidden behind a marching band, then pops out, then gets obscured by a tree. This is the challenge of Point Tracking in computer vision: keeping your eye on a single dot as it moves through a video.

Most computer programs that do this today are like students who have memorized a textbook. They were trained on millions of labeled videos (where humans drew dots on every frame). They are good, but if they see a parade they haven't seen before, or if the lighting changes, they get confused. They need a lot of expensive homework (training data) to learn.

This paper introduces a new method called HeFT (Head-Frequency Tracker). Instead of memorizing a textbook, HeFT uses a super-intelligent artist who has never seen your specific parade but has watched every parade ever made.

Here is how it works, broken down with simple analogies:

1. The "Super-Artist" (The Video Diffusion Model)

The researchers use a pre-trained AI called a Video Diffusion Transformer (VDiT). Think of this AI as a master painter who has spent years learning how to generate realistic videos from scratch. Because it learned to create videos, it understands how objects move, how they look, and how they relate to each other over time. It has "common sense" about the world.

The team asked: Can we use this artist's brain to track a balloon, even though the artist was never taught to track things?

2. The "Orchestra" Analogy (Head Specialization)

Inside this super-intelligent AI, there isn't just one brain; there is a massive orchestra of Attention Heads.

• The Old Way: Previous methods treated the whole orchestra as one big, blurry sound. They took the average of everything.
• The HeFT Discovery: The researchers realized that each musician (head) has a special job.
  • Some musicians are Matchmakers: They are experts at finding "that's the same balloon from the last frame!"
  • Some are Semantics: They care about the type of object (e.g., "that's a person").
  • Some are Positional: They care about where things are in space.

The Analogy: Imagine trying to find a friend in a crowd. If you ask a whole group of people (the whole layer) for help, you get a confused mumble. But if you ask the one person who is an expert at recognizing faces (the Matching Head), they will point you right to your friend. HeFT ignores the noise and only listens to the "Matchmaker" musician.

3. The "Radio Static" Analogy (Frequency Filtering)

The AI processes information in different "frequencies," kind of like radio waves.

• Low Frequencies: These are the smooth, steady signals. They tell you the big picture: "The balloon is moving left." They are stable and reliable.
• High Frequencies: These are the sharp, jagged signals. They contain tiny details like texture, noise, or static. While they look cool, they are often noise that confuses the tracker.

The Analogy: Imagine trying to hear a conversation in a noisy room. The "High Frequencies" are the clinking of silverware and the chatter of other tables. The "Low Frequencies" are the clear voice of your friend. HeFT puts on noise-canceling headphones to block out the high-frequency static and only listens to the smooth, low-frequency voice that tells the truth about where the object is.

4. The "One-Step" Trick (Denoise to Track)

Usually, these AI models take many steps to turn a blurry mess into a clear video. The researchers found a shortcut. They realized that if they take a real video, add a tiny bit of "noise" to it, and then let the AI clean it up just once, the AI's internal "Matchmaker" neurons light up with the perfect information needed to track the point.

It's like asking the artist to "fix" a slightly blurry photo for a split second. In that split second, the artist's brain reveals exactly where the object is, without needing to finish painting the whole picture.

5. The "Safety Net" (Forward-Backward Check)

To make sure the tracker doesn't get lost when the balloon hides behind a tree, HeFT uses a safety net.

• It tracks the balloon forward in time.
• Then, it tracks backward from the new spot to the start.
• If the two paths don't meet up perfectly, the system knows the balloon is hidden (occluded) and stops guessing, preventing it from drifting off into the wrong part of the screen.

The Result

By listening to the right "musician" (Head Selection) and filtering out the "static" (Frequency Filtering), HeFT can track points in videos without any training data.

• Zero-Shot: It hasn't been taught with labeled dots. It just uses its general knowledge of how the world works.
• Performance: It performs almost as well as the super-expensive, heavily trained systems, but it's much more robust and doesn't need a massive dataset.

In short: The paper teaches us that we don't need to train a new specialist for every job. We can just take a generalist genius (the Video Diffusion Model), ask the right specific question (select the right head), and ignore the noise (filter the frequencies) to get world-class results.

Technical Summary

1. Problem Statement

Point tracking involves estimating the temporal correspondence of physical points across video frames. Current state-of-the-art methods are predominantly supervised, relying on large-scale, costly, and time-consuming annotated datasets (e.g., TAP-Vid). These supervised models often struggle with domain shifts and lack robustness when applied to unseen data.

While Video Diffusion Transformers (VDiT) have demonstrated exceptional capabilities in generating temporally coherent videos, implying they possess rich visual priors of the real world, their potential as foundation models for perception tasks (specifically zero-shot point tracking) remains underexplored. Existing attempts to repurpose VDiTs for tracking often treat them as "black boxes," failing to systematically analyze how spatiotemporal information is encoded at the granular level of attention heads and frequency components.

2. Methodology: HeFT (Head-Frequency Tracker)

The authors propose HeFT, a zero-shot point tracking framework that leverages the internal representations of pretrained VDiTs. The core methodology is built upon two key insights derived from a systematic analysis of VDiT internals:

A. Head-Level Specialization

The authors analyzed the internal representations of VDiT and discovered that attention heads act as minimal functional units with distinct specializations, rather than the entire layer being the optimal unit for feature extraction. They identified three primary types of heads:

1. Matching-oriented heads: Capture precise correspondences across frames (diagonal/off-diagonal attention patterns).
2. Semantic-oriented heads: Attend to patches with similar semantic content (diffuse patterns).
3. Position-oriented heads: Focus on spatially adjacent patches (single main diagonal).

Strategy: Instead of aggregating features from all heads in a layer, HeFT performs Head Selection to identify and utilize the single most effective "matching-oriented" head for a given backbone, significantly reducing noise and redundancy.

B. Frequency-Aware Feature Filtering

VDiTs utilize 3D Rotary Positional Embeddings (RoPE) to encode spatiotemporal positions. The authors analyzed the frequency components of these embeddings and found:

• High-frequency components: Exhibit rapid rotation with positional changes, making them sensitive to fine-grained positional shifts but prone to introducing noise in correspondence matching.
• Low-frequency components: Remain nearly invariant across positions, providing stable cues for establishing correspondences.

Strategy: HeFT employs Frequency Filtering to retain only the low-frequency components (specifically the lowest ~45-50%) of the feature vectors. This suppresses high-frequency noise while preserving the stable matching signals.

C. The Tracking Pipeline

1. Feature Extraction: The method perturbs real-world videos with noise corresponding to the final denoising step of the diffusion process and performs a single-step denoising to extract features. This step was found to yield the most reliable perceptual cues.
2. Feature Selection: The pipeline selects the optimal attention head and filters out high-frequency components based on the insights above.
3. Correspondence Estimation:
   • Soft-Argmax: Trajectories are localized by applying a soft-argmax operation to correlation maps for robust sub-pixel localization.
   • Feature Refinement: Query features are progressively updated using a moving-average scheme to mitigate appearance drift over time.
   • Visibility Check: A forward-backward consistency check is employed to detect occlusions. If the deviation between forward and backward trajectories exceeds a threshold, the point is marked as occluded, preventing chaotic trajectory generation.

3. Key Contributions

1. Systematic Analysis of VDiT: The paper provides the first detailed analysis revealing that attention heads in VDiTs exhibit cooperative specialization (matching, semantic, positional) and that low-frequency components are crucial for correspondence, while high-frequency components introduce noise.
2. Novel Zero-Shot Framework (HeFT): Introduces a tracking framework that explicitly leverages these findings through a head- and frequency-aware feature selection strategy, eliminating the need for annotated training data.
3. State-of-the-Art Performance: Demonstrates that HeFT achieves performance comparable to fully supervised methods on standard benchmarks, bridging the gap between generative foundation models and discriminative perception tasks.

4. Experimental Results

The method was evaluated on TAP-Vid (DAVIS, Kinetics) and PointOdyssey benchmarks.

• Quantitative Performance:
  • HeFT significantly outperforms all existing zero-shot and self-supervised baselines (e.g., DIFT, SD-DINO, DiffTrack).
  • It approaches the accuracy of supervised methods (e.g., TAPIR, CoTracker3). For instance, on TAP-Vid DAVIS, HeFT (using Cosmos-Predict2-2B) achieved an Average Jaccard (AJ) of 48.61 and Occlusion Accuracy (OA) of 82.47, surpassing the supervised TAP-Net and approaching CoTracker3.
  • The method shows strong generalization across different VDiT backbones (CogVideoX-2B, Wan2.1-1.3B, Cosmos-Predict2-2B).
• Qualitative Results:
  • Visual comparisons show HeFT maintains stable trajectories in challenging scenarios (occlusions, fast motion, appearance changes) where other zero-shot methods fail or produce chaotic paths.
  • Ablation studies confirm that removing either head selection or frequency filtering causes a significant drop in performance, validating the necessity of both components.

5. Significance and Future Work

• Foundation Models for Perception: The work underscores the potential of video diffusion models not just as generative tools but as powerful foundation models for a wide range of downstream perception tasks.
• Unified Visual Understanding: It paves the way for unified visual foundation models that can handle both generation and discrimination tasks without task-specific fine-tuning.
• Limitations & Future Directions:
  • Offline Processing: Currently requires processing the entire video (or chunks) through the diffusion model, limiting real-time application. Future work aims to explore distillation for online, streaming tracking.
  • Computational Cost: High-resolution processing requires significant GPU memory. Future research may focus on memory-efficient feature extraction strategies.

In conclusion, HeFT successfully "denoises" the internal representations of video diffusion models to "track" points, proving that carefully extracted priors from generative models can rival supervised learning in point tracking tasks.