Technical Summary: G2TAM (Geometry Grounded Track Anything Model)

Problem Statement

Current video object segmentation (VOS) and instance tracking methods predominantly rely on appearance matching and explicit temporal memory banks. While effective in static or slow-moving scenarios, these approaches struggle with drastic viewpoint changes, long-term occlusions, and complex dynamic scenes. They often fail to maintain object identity when visual cues are degraded or when the object moves out of view.

Existing attempts to incorporate 3D geometry typically follow a "Tracking-by-Mapping" paradigm, where 2D tracking results are post-hoc projected into 3D space using off-the-shelf pose and depth estimators. This multi-stage pipeline suffers from error propagation between disconnected modules and is strictly limited to scenarios where accurate geometry is already available. The paper posits a fundamental question: Can geometry serve not merely as a post-hoc verification tool, but as a "latent foundation" that implicitly grounds the entire perception process?

Methodology

The authors propose G2TAM (Geometry Grounded Track Anything Model), a unified, end-to-end framework for promptable instance tracking in 3D space using only unordered RGB images or videos. The core innovation is the concept of Geometry as Implicit Memory, where spatially aligned geometric representations replace explicit temporal memory banks to ensure stable instance identity and localization.

Architecture

G2TAM is built upon the Pi3 (Permutation-equivariant visual geometry learning) architecture and consists of three main components:

1. Prompt Encoders:
   • Visual Prompts: Points and boxes are encoded using positional encoding (random Fourier features) and learnable type embeddings, similar to SAM.
   • Text Prompts: Text is tokenized and processed through a CLIP text encoder, projected into a learnable embedding.
2. Cross-Modal Spatial Encoder:
   • This is the core innovation. It fuses visual tokens (from a DINOv2 backbone), prompt tokens, and learnable register tokens into a unified sequence.
   • Early Fusion: Unlike methods that inject prompts late in the decoding stage, G2TAM integrates text and visual prompts early. Text tokens are inserted as global conditioning signals across all frames, while visual prompts are localized to specific frames.
   • The encoder utilizes alternating intra-view self-attention and global cross-view attention blocks to construct a shared geometric-semantic feature space.
3. Geometry & Mask Decoders:
   • Geometry Decoder: Inherited from Pi3, it predicts a comprehensive geometry set for each frame, including camera pose ($K_i$), pixel-aligned 3D point maps ($X_i$), and a confidence map ($C_i$).
   • Mask Decoder: Adapted from SAM, it predicts instance masks ($M_i$). Due to the early fusion, prompt embeddings are eliminated during mask decoding; the model relies on the fused geometric-semantic representation. An additional MLP head predicts the likelihood of the target object's presence.

Training Strategy

The model is trained end-to-end on a composite loss function:
$$L = \lambda_{seg}L_{seg} + \lambda_{geo}L_{geo}$$

• Segmentation Loss ($L_{seg}$): Weighted sum of Binary Cross-Entropy and Dice loss.
• Geometry Loss ($L_{geo}$): Includes point reconstruction loss, surface normal loss, confidence loss, and camera pose loss.
• Data: The model is trained on a large-scale aggregation of segmentation datasets (e.g., DAVIS, SA-V), reconstruction datasets (e.g., ScanNet++, OmniWorld), and the newly introduced InsTrack dataset.

Dataset: InsTrack

To support training and evaluation, the authors constructed InsTrack, a large-scale dataset derived from ScanNet++. It features:

• 856 training and 50 validation scenes.
• Multi-modal prompts (text, points, boxes) and instance-consistent 3D masks.
• A dedicated validation split for benchmarking.
• An automated pipeline for generating prompts and rasterizing 3D mesh annotations to 2D masks.

Key Contributions

1. G2TAM Framework: A unified model that performs joint 3D reconstruction and spatial-temporal consistent instance segmentation without explicit 3D inputs or temporal memory banks.
2. Geometry as Implicit Memory: The proposal that spatially aligned geometric representations can serve as a persistent memory for identity reasoning, outperforming appearance-based memory in challenging views.
3. Cross-Modal Spatial Encoder: An early-fusion mechanism that integrates text and visual prompts directly into a shared geometric space, enabling robust multi-modal reasoning.
4. InsTrack Dataset & Benchmark: A new dataset and the Promptable Instance Spatial Tracking (PIST) task, designed to evaluate instance segmentation consistency across views in static scenarios.

Experimental Results

Extensive experiments demonstrate G2TAM's superiority over state-of-the-art methods:

• Promptable Instance Spatial Tracking (PIST): On the InsTrack validation set, G2TAM achieves 74.3 S-mIoU, significantly outperforming SAM2 (47.6 S-mIoU) and other VOS models. It maintains high consistency even under large viewpoint changes where appearance-based models fail.
• 3D Visual Grounding: G2TAM achieves 45.7% Acc@0.5 on ScanRefer, surpassing the previous SOTA (VLM-Grounder, 32.8%). It demonstrates that the model can infer object relationships directly from 2D images without relying on explicit point cloud inputs.
• Video Object Segmentation:
  • Semi-supervised VOS: G2TAM outperforms SAM2 on benchmarks like MOSE (77.8 vs 75.2 J&F), showing robustness to heavy occlusion.
  • Referring VOS (RVOS): G2TAM achieves 72.2 J&F on Ref-YTVOS, outperforming ReferDINO by +2.9.
• Ablation Studies:
  • Joint Training: Jointly training reconstruction and segmentation yields a 4.0% improvement in segmentation accuracy compared to segmentation-only training, validating geometry as a regularizer.
  • Memory: Adding explicit memory modules (like SAM2) does not consistently improve performance, suggesting that the implicit geometric memory is sufficient.
  • Temporal Robustness: The model maintains performance even when input frames are temporally shuffled, confirming reliance on spatial consistency rather than temporal continuity.

Significance and Claims

The paper claims that G2TAM establishes a foundation for interactive, geometry-grounded spatial reasoning. By unifying appearance and geometry within a latent, spatially-aligned feature space, the model achieves robust tracking without the error propagation inherent in multi-stage pipelines.

The authors emphasize that their approach enables:

• Stable Identity: Maintaining object identity across drastic viewpoint shifts and long-term occlusions.
• Multi-Modal Interaction: Supporting text, point, and box prompts for 3D instance tracking.
• Generalization: Performing well across static and dynamic scenarios, as well as in 3D visual grounding tasks, without requiring explicit 3D geometry inputs at inference time.

The work suggests a shift from "Tracking-by-Mapping" to a paradigm where geometry is an intrinsic, implicit component of the perception process, offering a more robust solution for spatial understanding in complex environments.