Technical Summary: Probing and Leveraging Video Diffusion Transformer Features for Robust Point Tracking

1. Problem Statement

Current point tracking methods, while effective on standard benchmarks, often rely on feature backbones (e.g., ResNet) that lack inherent temporal coherence, leading to poor performance in real-world scenarios involving motion blur, fast motion, and frequent occlusions. Although recent works have begun incorporating Visual Foundation Models (VFMs) like DINOv2, there has been no systematic analysis determining which VFM architecture or training objective yields the most robust representations for dense temporal tracking. Furthermore, existing approaches typically require extensive supervision on large-scale real-world video datasets to achieve high performance.

2. Methodology

2.1. Systematic Analysis of VFMs

The authors conducted a zero-shot evaluation of diverse VFMs to identify the most robust backbone for point tracking. The study compared models across:

• Pretraining Modalities: Image-only vs. Video.
• Architectures: UNet vs. Transformer.
• Learning Objectives: Self-supervised (MAE, DINO) vs. Diffusion.
• Benchmarks: Standard (TAP-Vid-DAVIS), Motion Blur corrupted (TAP-Vid-DAVIS + ImageNet-C), and Challenging Real-World (ITTO-MOSE).

Key Findings from Analysis:

• Video Diffusion Transformers (DiTs) consistently outperformed all other VFMs, including image-trained models (DINOv2, SD3) and video models with decomposed attention (SVD) or MAE objectives (V-JEPA).
• CogVideoX-5B achieved the highest performance, surpassing even ResNet backbones explicitly supervised on tracking data.
• Attributed Factors: The superiority of Video DiTs is attributed to three characteristics:
  1. Large-scale real-world video pretraining: Exposure to diverse motion and appearance variations.
  2. Full 3D spatio-temporal attention: Enables global interaction across space and time, unlike decomposed attention mechanisms.
  3. Diffusion training objective: Reconstructs all tokens across noise levels, producing richer correspondence representations compared to spatial-only objectives like MAE.
• Scaling Law: Within the same model family, larger models (e.g., WAN-14B vs. WAN-1.3B) consistently yielded better tracking performance, particularly under challenging conditions.

2.2. DiTracker Framework

Motivated by the analysis, the authors propose DiTracker, a framework that integrates Video DiT features into existing tracking pipelines (specifically using the CoTracker3 head) through three core components:

1. Query-Key Matching Cost Computation: Instead of using hidden states, DiTracker extracts query ($Q$) and key ($K$) projections directly from the Video DiT's full 3D attention layers. These projections inherently encode inter-frame correspondences. A matching cost is computed via scaled dot-product attention:
   $$C_{DiT} = \text{Softmax}\left(\frac{q_i k_j^\top}{\sqrt{d_{head}}}\right)$$
2. Cost-Level Fusion: Since Video DiTs operate in a spatially compressed latent space (e.g., $H/16 \times W/16$), they lack fine-grained spatial details required by standard tracking heads. DiTracker fuses the DiT matching cost with a lightweight ResNet branch that provides higher-resolution features. The costs are flattened, concatenated, and projected into a cost embedding via an MLP:
   $$E_j = \text{MLP}([\text{Flatten}(C_{DiT}), \text{Flatten}(C_{ResNet})])$$
   This preserves the DiT's pretrained matching distribution while recovering spatial details.
3. LoRA Adaptation: To transfer generative features to the tracking task efficiently, the authors employ Low-Rank Adaptation (LoRA). Trainable low-rank matrices are inserted into the attention layers of the frozen Video DiT backbone. The model is trained with a combined objective of trajectory regression, visibility classification, and confidence losses.

3. Key Contributions

1. First Systematic Analysis: The paper presents the first comprehensive study of VFM representations for robust point tracking, identifying Video DiTs as the most robust backbone, even outperforming supervised ResNet backbones.
2. DiTracker Framework: Introduction of a method to leverage Video DiT features for supervised tracking via query-key matching, cost-level fusion with a ResNet branch, and LoRA adaptation.
3. Data Efficiency and Robustness: Demonstration that DiTracker, trained solely on synthetic data (Kubric) with significantly fewer iterations, outperforms CoTracker3 (which is trained on additional real-world videos) under the same tracking head. It shows the largest performance gains in corrupted and challenging scenarios.
4. Generalization and Scalability: Validation that the benefits of Video DiT features generalize across different tracking heads (e.g., AllTracker) and scale with backbone size, confirming Video DiTs as a robust, data-efficient, and head-agnostic backbone.

4. Experimental Results

• Standard Benchmarks: On TAP-Vid-DAVIS and ITTO-MOSE, DiTracker (trained on synthetic data only) surpassed CoTracker3 trained with 15k real-world videos. For instance, on ITTO-MOSE, DiTracker improved $\delta^x_{avg}$ by 2.7% over CoTracker3.
• Robustness Benchmarks: Under ImageNet-C corruptions (noise, blur, weather), DiTracker achieved the best average score (71.1%), outperforming CoTracker3 (69.2%) despite the latter's real-world training.
• Ablation Studies:
  • Feature Selection: Query-key projections (52.6% AJ) significantly outperformed hidden states (45.7% AJ).
  • Fusion Strategy: Concatenating DiT and ResNet costs (72.2% $\delta^x_{avg}$) outperformed simple upsampling or feature concatenation.
  • LoRA: Adaptation was essential; frozen DiT features yielded poor results (41.0% AJ). Full-layer LoRA with higher ranks (128) provided the best performance.
• Scaling: Larger backbones (WAN-14B) consistently outperformed smaller variants (WAN-1.3B) across all metrics.

5. Significance and Claims

The paper claims that generative video pretraining provides strong real-world priors for correspondence, effectively reducing the dependence on large-scale real-data supervision for point tracking. By leveraging Video DiTs, the authors demonstrate that:

• The temporal coherence required for robust tracking is inherent in generative video models trained with diffusion objectives and full 3D attention.
• It is possible to achieve state-of-the-art robustness using only synthetic data and fewer training iterations compared to methods relying on real-world video supervision.
• Future advances in video generation can directly benefit point tracking tasks, suggesting a convergence between generative and discriminative video understanding.