Real-World Point Tracking with Verifier-Guided Pseudo-Labeling

1. Problem Statement

Long-term point tracking is a fundamental task in computer vision, essential for applications like video editing, robotics, and 4D scene understanding. While transformer-based trackers have achieved state-of-the-art (SOTA) performance, they face a critical bottleneck:

• Sim-to-Real Gap: Models are typically trained on large synthetic datasets (e.g., TAP-Vid Kubric) because dense, frame-accurate ground-truth annotations for real-world videos are prohibitively expensive to create.
• Performance Degradation: When deployed on real-world videos, these models suffer from domain shifts involving appearance statistics, non-rigid motion, occlusion, and lighting changes.
• Limitations of Naïve Self-Training: A common solution is self-training using pseudo-labels generated by pre-trained "teacher" models on unlabeled real data. However, this approach is brittle because:
  • No single teacher is reliable across all frames or scenes.
  • Different trackers fail in different ways (e.g., one drifts on low texture, another suffers identity switches on occlusion).
  • Fixed heuristics (e.g., random selection or averaging) cannot dynamically reconcile these heterogeneous error profiles, leading to error accumulation and model collapse.

Core Challenge: How to effectively fine-tune point trackers on unlabeled real-world videos by generating high-quality pseudo-labels that adaptively select the most reliable predictions from multiple diverse teachers.

2. Methodology

The authors propose Verifier, a learned meta-model designed to assess the reliability of tracker predictions and guide pseudo-label generation. The framework operates in two main phases:

A. The Verifier Architecture

The Verifier is a meta-model trained entirely on synthetic data with ground-truth trajectories. It does not predict trajectories directly but learns to score the reliability of candidate trajectories provided by multiple off-the-shelf trackers.

1. Input:
   • A query point $q_{t_0}$ in a video.
   • A set of $M$ candidate trajectories (predictions from different teacher trackers).
2. Feature Extraction (Localized Context):
   • The model extracts dense visual features using a frozen CNN encoder (from CoTracker3).
   • It samples features at the query location (reference) and at the predicted locations of all candidates.
   • Deformable Attention: It aggregates local context around each candidate relative to the query's initial appearance, capturing spatial consistency.
   • Embeddings: It incorporates positional embeddings (displacement from the query) and identity embeddings (distinguishing the query from candidates).
3. Candidate Transformer:
   • A specialized transformer decoder processes the query and candidate features.
   • Cross-Attention: At each frame, the query attends to all candidate features to identify the most consistent motion hypothesis.
   • Temporal Self-Attention: It connects frames over time to resolve ambiguities (e.g., if a frame is uncertain, context from confident neighboring frames helps).
4. Output:
   • A per-frame reliability distribution ($\hat{s}_t$) over the $M$ candidates, computed via a temperature-scaled softmax on cosine similarities.

B. Training Strategy (Synthetic Domain)

• Data: Trained on K-EPIC (synthetic dataset).
• Candidate Generation: Ground-truth trajectories are perturbed with stochastic transformations to simulate realistic errors (drift, jitter, occlusion, identity switches, abrupt jumps).
• Objective: A soft contrastive loss. The target reliability distribution is a softmax over the negative Euclidean distance between candidates and the ground truth. The Verifier learns to rank correct trajectories higher than corrupted ones.

C. Real-World Adaptation Pipeline

1. Pseudo-Label Generation:
   • For an unlabeled real video, multiple teacher trackers generate candidate trajectories.
   • The Verifier evaluates these candidates frame-by-frame.
   • The candidate with the highest reliability score is selected as the pseudo-label for that frame.
   • Visibility is determined by majority voting among teachers.
2. Fine-Tuning:
   • The base tracker (Track-On2) is fine-tuned on real-world data using these Verifier-guided pseudo-labels.
   • Curriculum Schedule: The training mixes synthetic data (ground truth) and real data (pseudo-labels), gradually increasing the weight of real-world samples to shift the model's distribution toward the real domain.

3. Key Contributions

1. The Verifier Meta-Model: A learnable module that scores frame-wise reliability of multiple trackers, enabling dynamic selection of the best prediction rather than relying on a single teacher or fixed heuristics.
2. Verifier-Guided Pseudo-Labeling: A framework that transforms model diversity into a strength. By adaptively fusing predictions based on learned reliability cues, it generates high-quality supervision for unlabeled videos without manual annotation.
3. Inference-Time Ensemble: The Verifier can also be used at inference time as a plug-and-play ensemble module, selecting the best tracker per frame to improve robustness without retraining.
4. Data Efficiency: The approach achieves SOTA results with significantly less data than prior self-training methods that rely on massive-scale distillation or specific domain priors.

4. Experimental Results

The method was evaluated on four real-world benchmarks: TAP-Vid DAVIS, TAP-Vid Kinetics, RoboTAP, and EgoPoints.

• Performance:
  • Track-On-R (the fine-tuned model) achieved SOTA results across all datasets.
  • On EgoPoints (challenging ego-centric videos), it improved $\delta_{avg}$ from 61.7 (synthetic baseline) to 67.3, outperforming the closest competitor (AllTracker) by +5.3.
  • On RoboTAP, it achieved the highest scores, despite the training data containing no robotic sequences, demonstrating strong generalization.
  • On Kinetics and DAVIS, it outperformed all synthetic baselines and other real-world fine-tuning methods (e.g., BootsTAPIR, CoTracker3).
• Ablation Studies:
  • Teacher Composition: The Verifier remains robust even when weaker teachers are added to the ensemble, effectively exploiting complementary strengths.
  • Non-Learning Baselines: The learned Verifier significantly outperforms fixed heuristics like geometric median, agreement-based selection, and Kalman filters, proving that rigid rules cannot capture diverse failure modes.
  • Data Efficiency: Fine-tuning on only 3,000 real videos (TAO dataset) yielded results comparable to using the full collection (4,800 videos), highlighting the method's efficiency.

5. Significance

This paper addresses a critical bottleneck in computer vision: the lack of annotated real-world data for long-term tracking.

• Paradigm Shift: It moves away from "one model fits all" or "random ensemble" approaches to a learned, adaptive selection mechanism.
• Practicality: It enables the adaptation of powerful synthetic-trained models to the real world using only raw, unlabeled video, making high-quality point tracking accessible for diverse applications (robotics, medical analysis, AR/VR) without the cost of manual annotation.
• Generalizability: The Verifier concept is not limited to point tracking; it offers a general framework for model coordination and uncertainty estimation in video correspondence tasks.

In summary, the authors demonstrate that by learning when and where to trust different trackers, one can construct a robust, data-efficient pipeline that bridges the sim-to-real gap and achieves state-of-the-art performance in real-world point tracking.