Technical Summary: Gaga: Group Any Gaussians via 3D-aware Memory Bank

Problem Statement

Effective open-world 3D segmentation is critical for scene understanding and manipulation. While zero-shot 2D class-agnostic segmentation models (e.g., SAM, EntitySeg) have advanced significantly, extending these to 3D scenes presents a fundamental challenge: mask label inconsistency across multi-view images. Since 2D models process each view independently, the same object may receive different mask IDs in different views. Naively lifting these inconsistent masks to 3D results in ambiguous and fragmented segmentation.

Existing solutions attempt to resolve this via:

1. Contrastive Learning: Learning feature vectors for 3D Gaussians to cluster them. However, these methods often fail to assign unique, consistent mask IDs to segmentation groups across views.
2. Video Object Tracking: Treating multi-view images as a video sequence to track objects. This approach relies on the assumption of continuous, minimal view changes, which often fails in real-world scenarios with sparse inputs, significant camera pose variations, or similar/occluded objects.

Methodology: Gaga

Gaga proposes a framework to reconstruct and segment open-world 3D scenes by leveraging inconsistent 2D masks through a novel 3D-aware Memory Bank. The method operates without assuming continuous view changes or requiring user intervention.

1. Scene Reconstruction and Initial Masking

• 3D Reconstruction: The pipeline begins by reconstructing the 3D scene using Gaussian Splatting from a set of posed RGB images.
• 2D Mask Generation: Open-world segmentation models (e.g., SAM, EntitySeg) generate class-agnostic 2D masks for each input view.

2. 3D-Aware Memory Bank and Mask Association

The core innovation is the 3D-aware Memory Bank, which groups 3D Gaussians and assigns universal group IDs to 2D masks across views based on geometric overlap rather than temporal tracking.

• Corresponding Gaussians with Depth Guidance: For each 2D mask, the system identifies the corresponding 3D Gaussians. To ensure accuracy and avoid including background Gaussians that project into the mask but belong to distant objects, the method employs depth guidance. It renders a depth map, filters outliers using interquartile range (IQR) statistics, and selects only Gaussians within the valid depth range of the foreground object.
• Group ID Assignment via Gaussian Overlap: The system iteratively builds the memory bank:
  • For a new view's mask, it calculates the overlap between the mask's corresponding Gaussians ($G(m)$) and existing groups ($G_i$) in the memory bank.
  • Overlap is defined as the ratio of shared Gaussians to the total Gaussians in the mask: $Overlap(m, i) = \#(G(m) \cap G_i) / \#G(m)$.
  • If the overlap with an existing group exceeds a threshold (0.1), the mask is assigned that group's ID, and non-overlapping Gaussians are added to the group.
  • If no existing group has sufficient overlap, a new group ID is created.
  • This process ensures that masks corresponding to the same 3D object across different views receive the same universal ID.

3. Identity Encoding and Rendering

Once consistent group IDs are established, the system learns an identity encoding (a 16-dimensional feature vector) for each 3D Gaussian.

• Supervision: The learned feature vectors are splatted to 2D images and decoded via a linear layer and SoftMax to predict segmentation maps.
• Loss: The predicted maps are compared against the pseudo-ground truth masks generated by the 3D-aware memory bank (which contain consistent group IDs) using cross-entropy loss.
• Output: The trained model can render multi-view consistent segmentation masks and enable 3D scene manipulation (e.g., object removal, color changing, translation) by selecting Gaussians based on their identity encoding.

Key Contributions

1. Framework for Inconsistent 2D Masks: A method to reconstruct and segment 3D scenes using inconsistent 2D masks from open-world models without requiring user guidance.
2. 3D-Aware Memory Bank: A novel mechanism that resolves mask inconsistency by categorizing 3D Gaussians and aligning 2D masks across diverse views based on 3D geometric overlap, eliminating the need for video tracking assumptions.
3. Versatility: The method is compatible with any 2D class-agnostic segmentation model, facilitating novel-view image and mask synthesis.
4. Robustness: Demonstrated effectiveness in challenging scenarios, including sparse input views and significant camera pose variations.

Experimental Results

The paper evaluates Gaga on multiple datasets (LERF-Mask, Replica, ScanNet, MipNeRF 360) against state-of-the-art methods including Gaussian Grouping, OmniSeg3D, and LangSplat.

• Quantitative Performance:
  • On LERF-Mask, Gaga achieves superior mIoU (92.3%) and mBIoU (90.8%) compared to previous methods.
  • On Replica and ScanNet, Gaga consistently outperforms Gaussian Grouping in IoU, Precision, and Recall, regardless of the 2D segmentation backbone (SAM or EntitySeg).
  • Sparse Data Robustness: When trained with only 5% of the data, Gaga significantly outperforms Gaussian Grouping (e.g., 31.87% vs. 21.76% IoU on Replica with SAM), demonstrating greater resilience to data scarcity.
• Qualitative Performance:
  • Gaga produces multi-view consistent masks with fewer artifacts, fewer empty regions, and sharper boundaries compared to contrastive learning and video tracking baselines.
  • It successfully handles similar objects and occlusions where video trackers fail (e.g., misidentifying similar sofas or losing objects during large camera movements).
• Applications: The paper demonstrates successful scene manipulation tasks, including precise object removal, color changing (e.g., recoloring only a cushion without affecting the footstool), and object translation, highlighting the accuracy of the 3D grouping.

Significance and Claims

The authors claim that Gaga addresses a fundamental limitation in existing open-world 3D segmentation: the inadequate exploitation of inherent 3D scene information to resolve 2D mask inconsistencies. By shifting from temporal tracking or contrastive learning to geometry-based cross-view mask association via a 3D-aware memory bank, Gaga achieves:

• True Multi-View Consistency: Assigning universal group IDs to objects across diverse camera poses without assuming continuous view changes.
• Robustness to Sparsity: Maintaining performance even with sparse input views where video tracking fails.
• Practical Utility: Enabling high-quality downstream applications such as 3D scene understanding and precise scene manipulation.

The paper concludes that these advancements make Gaga a robust solution for real-world 3D scene segmentation and editing tasks.