Technical Summary: SAGO – Online Segment 3D Gaussians via Launching Virtual Drones

1. Problem Statement

Interactive segmentation of 3D Gaussian Splatting (3DGS) scenes is hindered by the computational overhead of existing methods. Current state-of-the-art approaches typically require a time-consuming "setup stage" (ranging from tens of seconds to minutes) before interactive segmentation can occur. This setup involves multi-view mask preparation, mask lifting, feature distillation, and contrastive learning on a per-scene basis. These requirements create a significant bottleneck for online applications, preventing real-time manipulation and editing of raw 3DGS scenes. The core challenge addressed by this work is to eliminate this setup stage entirely while maintaining segmentation quality comparable to offline methods and achieving interaction latencies under one second.

2. Methodology: SAGO

The authors propose SAGO (Segment Any Gaussians Online), a setup-free framework that reframes 3D segmentation as an online Next-Best-View (NBV) planning task. The method introduces the concept of "virtual drones" to explore the scene dynamically.

Core Concept: Virtual Drones and NBV Planning

Instead of relying on pre-computed features or static training views, SAGO treats the segmentation process as a Markov decision process.

• Virtual Drones: Starting from the user's initial interaction view, the system launches virtual drones that iteratively plan their next viewing position (NBV) to maximize the coverage of the target object.
• Dual Trajectories: Two virtual drones are launched simultaneously from the interaction viewpoint, exploring the scene in clockwise and counter-clockwise directions (covering a 180-degree yaw range each).
• State Representation: The system state $S_t$ at each step is defined by a four-tuple:
  1. $\mathcal{A}^{(t)}$: The current set of foreground Gaussians.
  2. $\mathbf{v}_t$: The virtual drone's pose (target center $c_t$, radius $r_t$, yaw $\phi_t$, pitch $\theta_t$).
  3. $\mathcal{M}_t$: A memory bank for the Segment Anything Model (SAM) to track the object across views.
  4. $\tau_t$: A coverage indicator.

Technical Pipeline

1. State Initialization:

   • The process begins with a user prompt (click, box, or text) on an initial 2D view.
   • Mask-shaped Frustum Filtering (MFF): The initial 2D mask is used to filter the 3D Gaussian set. The authors employ center-based MFF, where a Gaussian is classified as background if its 3D center projects outside the 2D mask contour. This prunes the search space immediately.
   • The target center $c_t$ is initialized based on the 3D positions of Gaussians projected inside the initial mask.

2. Online NBV Planning:

   • The system employs an Exploration-Evaluation (EE) strategy to determine the optimal yaw ($\phi$) and pitch ($\theta$) angles for the next view.
   • The search space for yaw increments ($\Delta\phi$) is $\{90^\circ, 60^\circ, 30^\circ\}$, and for pitch ($\theta$) is $\{0^\circ, 30^\circ, 60^\circ\}$.
   • The NBV is selected to maximize the number of background Gaussians that can be filtered out ($|\Delta\mathcal{A}'|$) while ensuring the new segmentation mask maintains consistency with the initial mask (measured by mIoU $\geq \sigma$) to prevent cascading errors.
   • If a large yaw increment (e.g., $90^\circ$) fails to track the target, the system falls back to smaller increments ($60^\circ$, then $30^\circ$).

3. State Update:

   • The virtual drone navigates to the selected NBV, rendering a new view.
   • SAM (Segment Anything Model) generates a segmentation mask for this new view, utilizing the memory bank $\mathcal{M}_t$ for tracking.
   • MFF is applied again to the new view to update the foreground set $\mathcal{A}^{(t+1)}$ and background set $\mathcal{B}^{(t+1)}$.
   • The process iterates until full coverage of the target object is achieved.

3. Key Contributions

• Setup-Free Framework: SAGO completely eliminates the per-scene setup stage required by existing optimization-based methods, enabling interactive segmentation on raw 3DGS scenes.
• Virtual Drone Paradigm: The paper introduces a principled formulation that recasts 3D online segmentation as an NBV planning problem for virtual drones, leveraging active vision principles to resolve occlusions and refine boundaries.
• Efficiency and Quality: The method achieves a 50$\times$ speedup in inference time compared to previous setup-free frameworks (reducing latency from 40–60s to 0.4–0.9s) while delivering segmentation quality comparable to or superior to state-of-the-art offline methods.

4. Experimental Results

The authors evaluated SAGO on multiple datasets, including SPIn-NeRF, NVOS, 3D-OVS, LERF-Mask, and others.

• Quantitative Performance:
  • NVOS: Achieved 92.7% mIoU and 98.7% mAcc, outperforming both offline (e.g., SAGA) and online (e.g., iSegMan, GaussianCut) baselines.
  • SPIn-NeRF: Achieved 92.5% mIoU and 99.3% mAcc, ranking second in mIoU and first in mAcc among all compared methods.
  • 3D-OVS: Achieved a mean mIoU of 96.2%, surpassing the previous best (SAGA) by 0.2%.
  • LERF-Mask: Achieved an average mIoU of 91.0%, outperforming ClickGaussian by 1.9%.
• Latency: The average segmentation latency per interaction is approximately 500 ms (0.4–0.9s), enabling real-time interaction.
• Ablation Studies:
  • Removing the NBV strategy (relying only on original scene views) caused a dramatic performance drop, particularly in occluded scenes (e.g., a 54.3% drop in mIoU on LERF-Mask), validating the necessity of active view exploration.
  • Center-based MFF was found to be slightly more efficient and effective than splat-based MFF.

5. Significance and Claims

The paper claims that SAGO represents a significant shift in 3DGS interaction by removing the barrier of pre-computation. By leveraging virtual drones and NBV planning, the method enables:

• Direct Asset Extraction: Users can extract clean 3D assets from raw scenes in under one second.
• Online Editing: Extracted assets can be immediately manipulated (relocated, removed, copied, pasted) within the 3DGS scene.
• Broad Applicability: The method supports diverse prompts (clicks, boxes, text) and generalizes across various datasets without scene-specific tuning.

The authors acknowledge limitations, noting that center-based MFF may result in jagged boundaries (though this can be addressed by subsequent trimming methods) and that the method relies on a high-quality initial view for state initialization. Future work is suggested to explore active strategies for finding optimal initial views.