TAPNext++: What's Next for Tracking Any Point (TAP)?

1. Problem Statement

Tracking Any Point (TAP) is a fundamental computer vision task crucial for AR/XR and robotics, requiring the consistent tracking of arbitrary points across video frames despite occlusions, appearance changes, and long time horizons.

While the previous state-of-the-art model, TAPNext, introduced an efficient, purely online, recurrent transformer architecture capable of frame-by-frame tracking with minimal latency, it suffers from two critical limitations:

1. Long-Sequence Degradation: It struggles to maintain accurate tracks over long video sequences (e.g., >150 frames), as its recurrent state degrades over time.
2. Re-detection Failure: It frequently fails to re-detect query points after they have been occluded or have left the frame and re-entered. Current metrics often overlook this specific failure mode, focusing instead on average survival rates which include always-visible points.

Existing solutions to these problems (e.g., sliding windows, explicit memory banks, or re-detection via foundation models) often sacrifice the low-latency and low-memory efficiency required for mobile and real-time applications.

2. Methodology

TAPNext++ retains the core TAPNext architecture (a linear recurrent State-Space Model (SSM) combined with Vision Transformers) but introduces significant improvements in training strategies, datasets, and data augmentations to overcome the identified limitations.

A. Long-Sequence Training via Sequence Parallelism

To train the recurrent transformer on sequences significantly longer than the original 48-frame limit (up to 1024 frames), the authors employed Sequence Parallelism.

• Challenge: Standard training on 1024-frame sequences exceeds single-GPU memory limits due to the size of computation graphs and activations.
• Solution: They implemented a distributed parallel scan algorithm. The input sequence is sharded along the time dimension across multiple GPUs. Each GPU performs a local scan, followed by a logarithmic-time merge operation to combine hidden states across chunk boundaries. This allows end-to-end training on long sequences without breaking temporal context or requiring massive single-device memory.

B. New Datasets

The authors recognized the scarcity of long-sequence ground-truth data:

• Kubric-1024: A new synthetic dataset generated using the Kubric framework, consisting of 10,000 videos with 1024 frames each. It features dynamic objects, realistic lighting (Polyhaven HDRI), and complex camera trajectories.
• DynHumans: A synthetic dataset with 10,000 videos of 240 frames featuring human motion and camera movement.
• PointOdyssey: Used for fine-tuning, leveraging its existing long-sequence annotations.

C. Tailored Geometric Augmentations

To specifically address the re-detection blind spot, the authors introduced novel augmentations:

• Periodic Roll & Translation: Videos are rotated and translated smoothly. Crucially, pixels and track coordinates that shift off one boundary wrap around to the opposite side. This simulates points leaving the frame and re-entering from a different location, forcing the model to rely on appearance matching rather than just spatio-temporal proximity.
• Occlusion Supervision: The standard loss function only supervises visible points. TAPNext++ modifies the loss to include weighted supervision for occluded points (weight 0.2) that remain within image boundaries, encouraging the model to predict plausible locations during occlusion.

D. New Metric: Re-Detection Average Jaccard (AJRD)

The authors identified that standard metrics (like Survival Rate) do not adequately measure a model's ability to recover a track after a long period of invisibility.

• Definition: AJRD measures the tracking quality (Average Jaccard) specifically on track segments immediately following a reappearance event.
• Condition: It considers only "eligible" reappearance events where the duration of undetectability ($d$) sets a new record for that specific track.
• Usage: It is calculated for various minimum undetectability durations ($d_{min} \in \{1, 4, 16, 64, 256\}$) to evaluate robustness against short-term vs. long-term occlusions.

3. Key Contributions

1. TAPNext++ Model: A lightweight, fully online model that sets new state-of-the-art (SOTA) results on multiple benchmarks while maintaining the low memory/compute footprint of the original TAPNext.
2. Scalable Training: Demonstrated that recurrent video transformers can be effectively trained on 1024-frame sequences using distributed parallel scans, overcoming previous memory bottlenecks.
3. Kubric-1024 Dataset: The release of a large-scale synthetic dataset specifically designed for long-sequence point tracking training.
4. AJRD Metric: The introduction of a new evaluation metric to explicitly quantify re-detection performance, addressing a blind spot in current literature.
5. Geometric Augmentations: Proven effectiveness of periodic roll/translation augmentations in improving robustness to points re-entering the frame.

4. Results

TAPNext++ was evaluated on benchmarks including PointOdyssey, DAVIS, RoboTAP, Kinetics, and RGB-Stacking.

• Accuracy & Robustness:
  • Achieved SOTA Mean $\delta_{avg}$ of 72.2 across all benchmarks.
  • On PointOdyssey (long sequences): Improved Survival Rate to 67.9% (vs. 13.0% for BootsTAPNext) and reduced Median Translation Error (MTE) to 13.4.
  • On RoboTAP: Achieved SOTA AJRD scores, demonstrating superior ability to re-detect points after long occlusions.
• Speed & Efficiency:
  • Maintains high inference speeds: ~193 FPS (256x256) and ~348 FPS (1024x1024 windowed) on an NVIDIA H100 GPU.
  • Achieves the best speed-accuracy trade-off, outperforming competitors like CoTracker3 and Track-On while being significantly faster.
• Qualitative Performance:
  • Successfully tracks points in challenging scenarios where competitors fail, such as objects leaving the frame and re-entering from the opposite side, or tracking static points on texture-less surfaces during long sequences.

5. Significance

This work demonstrates that the limitations of recurrent architectures for long-term point tracking are not inherent to the model design but are solvable through better training data and strategies.

• Practical Impact: By achieving SOTA performance with low latency and memory usage, TAPNext++ makes robust, long-term point tracking feasible for real-time mobile applications, AR/VR, and robotics, where re-detection after occlusion is critical.
• Community Shift: The introduction of AJRD and the Kubric-1024 dataset provides the community with better tools to evaluate and train models for long-horizon and re-detection scenarios, moving beyond simple short-term tracking benchmarks.
• Architectural Validation: It validates that Linear Recurrent Units (SSMs) can effectively model long-term dependencies in video if trained correctly, without needing explicit memory banks or heavy offline processing.