Technical Summary: TrackRef3D

Problem Statement

Referring 3D Gaussian Splatting (R3DGS) aims to segment 3D objects using natural language queries, a capability critical for embodied AI, robotics, and autonomous driving. However, existing methods face three fundamental obstacles when scaling to real-world, open-world scenarios:

1. Reliance on Manual Annotation: Current pipelines require expensive, per-scene human-annotated category labels and referring descriptions, making deployment unrealistic.
2. Multi-View Inconsistency: Supervision is typically generated on a per-view basis using 2D foundation models. These models often fail under occlusion or with domain-specific vocabulary, generating contradictory pseudo-masks across different viewpoints.
3. Poor Generalization to Query Specificity: Models trained primarily on long, detailed referring descriptions tend to overfit, degrading performance on short, natural queries (e.g., single category names) that users frequently employ.

Methodology: TrackRef3D

TrackRef3D proposes a fully automatic pipeline that eliminates manual annotation by introducing a multi-view consistent track-then-label paradigm. This approach decouples object discovery from semantic grounding. The pipeline consists of three main stages:

1. Initial Object Discovery

The system processes unannotated RGB images using foundation models (Florence-2 for detection and SAM-2 for segmentation) to extract potential object masks and open-vocabulary category labels for each view. While this captures a wide range of objects, it inherently produces noisy masks (due to occlusion) and inconsistent labels (e.g., "bowl" vs. "ramen" for the same object across views).

2. Trajectory-Aware Semantic Consensus Module (TSCM)

To resolve multi-view inconsistencies, TSCM shifts the annotation focus from individual views to coherent object trajectories:

• Video Tracking: A tracker (DEVA) associates per-view masks into temporally coherent trajectories, assigning unique track IDs even when objects are occluded and reappear.
• Synonymous Clustering: Text embeddings of detected category labels are clustered using agglomerative clustering based on cosine similarity. This merges semantically equivalent labels (e.g., "cup" and "mug") into a single group.
• Trajectory-Aware Voting: For each trajectory, the system counts the frequency of clustered identities across all views. The most frequent identity is selected as the canonical semantic identity and propagated to all views in that trajectory, ensuring consistent supervision.
• Visibility-Aware Description Generation: To generate referring descriptions, the system selects a "keyframe" where the object is most visible. A visibility score balances mask area against the median area across the trajectory, penalizing extreme scales (too close/far) to ensure the selected view captures both object attributes and spatial context. A vision-language model (CogVLM-2) then generates multi-granularity descriptions based on this keyframe.

3. Hybrid Training Strategy (HTS)

To address the issue of overfitting to long descriptions, HTS employs a multi-positive contrastive objective.

• Instead of treating only long descriptions as positive pairs, the strategy jointly optimizes both the coarse category semantics (the canonical identity) and fine-grained referring descriptions for each track.
• All descriptions associated with the same trajectory are treated as positive pairs relative to the 3D Gaussian embeddings. This forces the model to learn robust representations that activate correctly for both short queries (category names) and long queries (detailed descriptions).
• The total loss combines this contrastive loss with a binary cross-entropy segmentation loss to ensure spatial accuracy.

Key Contributions

1. Fully Automatic Pipeline: TrackRef3D achieves open-world referring segmentation in 3DGS without any manual per-scene annotation.
2. TSCM for Consistency: The Trajectory-Aware Semantic Consensus Module aggregates cross-view predictions via synonymous clustering and voting to establish a canonical semantic identity, fundamentally solving multi-view inconsistency.
3. HTS for Robustness: The Hybrid Training Strategy ensures robustness under varying query specificities by jointly optimizing coarse and fine-grained cues, preventing the degradation of performance on short prompts.
4. State-of-the-Art Performance: The method demonstrates superior performance across multiple benchmarks compared to previous manual and automatic approaches.

Experimental Results

The authors evaluated TrackRef3D on four datasets: Ref-LERF, LERF-OVS, 3D-OVS, and a self-collected indoor Laboratory scene.

• Ref-LERF: TrackRef3D achieved an average mIoU of 38.8, outperforming the previous state-of-the-art (ReferSplat) by 9.6 points. It showed significant gains in specific scenes (e.g., +10.5 mIoU on the Ramen scene).
• Laboratory Scene: In a self-collected scene with no manual training labels, TrackRef3D achieved 48.5 mIoU on referring descriptions and 68.3 mIoU on short queries. Notably, ReferSplat (even when trained on the same automatic supervision) dropped to 24.4 mIoU on short queries, validating the necessity of HTS.
• Open-Vocabulary Benchmarks: On LERF-OVS and 3D-OVS, TrackRef3D achieved average mIoUs of 63.2 and 95.2 respectively, surpassing all compared methods including LangSplat and GOI.
• Ablation Studies: Experiments confirmed that removing TSCM or HTS significantly degrades performance. Specifically, the visibility-aware keyframe selection and the multi-positive contrastive loss were identified as critical components for handling occlusion and query variability.

Significance

The paper claims that TrackRef3D provides a scalable, open-world solution for referring segmentation in 3D Gaussian Splatting. By fundamentally decoupling object discovery from semantic grounding and enforcing cross-view consistency through trajectory tracking, the method eliminates the bottleneck of manual annotation. Furthermore, by addressing the generalization gap between short and long queries, it enables more intuitive and robust interaction for embodied AI systems in real-world environments.