Scene-Agnostic Object-Centric Representation Learning for 3D Gaussian Splatting

1. Problem Statement

Current approaches to 3D scene understanding, particularly those aiming for instance-level segmentation in 3D Gaussian Splatting (3DGS), rely heavily on Visual Foundation Models (VFMs) like SAM (Segment Anything Model) to generate 2D masks. These masks are then "lifted" to 3D to supervise the learning of object identity features. However, this paradigm suffers from three critical limitations:

1. View Inconsistency: VFM-generated masks often lack consistency across different viewpoints. The same physical object may receive different segmentation IDs at different angles, requiring complex pre/post-processing (e.g., tracking with DEVA) or specialized contrastive losses to align them.
2. Scene-Dependence: The learned object identities are tied to the specific scene's appearance. They do not generalize across different scenes, limiting applications like robotic interaction or cross-scene transfer.
3. Lack of True Object-Centricity: VFMs are trained for appearance-driven segmentation of arbitrary regions, not for reasoning about persistent objects. This leads to over-segmentation of single physical entities (splitting an object into parts based on surface cues) and fails to capture the underlying structure required for physical interaction.

Existing unsupervised Object-Centric Learning (OCL) methods based on Slot Attention (e.g., in NeRF) struggle with slot permutation (inconsistent slot IDs across views) and the implicit nature of NeRF, which hinders downstream applications.

2. Methodology

The authors propose a pipeline that integrates Unsupervised Object-Centric Learning (OCL) directly into 3DGS using a Global Object-Centric Learning (GOCL) framework. The core innovation is a scene-agnostic object codebook that anchors object identities globally, eliminating the need for per-scene fine-tuning or mask alignment.

Key Components:

• Backbone: GOLD (Global Object-centric Learning via Disentangled Slot Attention):

  • The authors adopt the pre-trained GOLD module, which learns a global codebook ($e_{glo}$) containing $C$ intrinsic object features.
  • GOLD disentangles intrinsic object properties (identity) from extrinsic factors (pose, lighting, scene context).
  • It uses Disentangled Slot Attention (DSA) to assign object slots to global prototypes via a Gumbel-Softmax mechanism, ensuring consistent identity selection across views and scenes without manual labels.

• 3DGS Integration:

  • Gaussian Identity Features: Each 3D Gaussian is assigned a learnable identity feature vector ($f_i$) of dimension $D_{code}$.
  • Rendering: Similar to color rendering, these identity features are projected to the 2D image plane using alpha-blending to produce a feature map $F_u$.

• Supervision Pipeline (Scene-Agnostic):

  1. Mask & Probability Generation: For a given view, the input image is passed through the trained GOLD module, which outputs $K$ object masks and a probability matrix $\gamma$ indicating the likelihood of each mask belonging to a specific global codebook entry.
  2. Code Assignment: For each mask, the codebook entry with the highest probability is selected ($c^*_k = \arg\max \gamma$). The corresponding global feature vector ($f^*_k$) is assigned to that mask.
  3. Dense Supervision: The selected features are broadcast to the mask resolution and blended to create a dense target feature map ($F^*_u$).
  4. Loss Function: The rendered Gaussian features are supervised directly against this target using an MSE loss ($L_{feature}$).
  5. 3D Regularization: A KL-divergence loss ($L_{3D}$) is applied to ensure spatially neighboring Gaussians share similar class probabilities, promoting coherent object grouping.

• Inference: Object IDs are assigned to pixels by finding the closest codebook feature via L2 distance, enabling immediate cross-scene identification without re-training.

3. Key Contributions

1. First Integration of Unsupervised OCL into 3DGS: The paper successfully bridges unsupervised object discovery with explicit 3D Gaussian representations, overcoming the slot permutation issues common in previous attempts.
2. Scene-Agnostic Object Codebook: Introduces a global codebook that provides consistent, identity-anchored representations across different views and entirely different scenes. This enables cross-scene identification without re-training.
3. Simplified Supervision Pipeline: The method eliminates the need for VFM-specific mask pre/post-processing (e.g., tracking, alignment) and avoids complex, ad-hoc training strategies or additional contrastive loss terms required by prior VFM-based methods.
4. Robustness to Over-segmentation: By relying on object-centric priors rather than surface appearance, the method produces more semantically meaningful decompositions of physical objects.

4. Experimental Results

The method was evaluated on two datasets: OCTA (real-world tabletop scenes) and GSO (synthetic objects).

• Quantitative Performance:

  • The proposed method significantly outperforms state-of-the-art VFM-based methods (Gaussian Grouping, Unified Lift, ObjectGS) on Object-Centric Learning (OCL) metrics (FG-ARI, FG-AMI, ARI-A, AMI-A).
  • On the OCTA dataset, it achieved an FG-ARI of 0.870 (vs. 0.700 for the next best) and FG-AMI of 0.868.
  • It maintains competitive mIoU scores, though slightly lower than some VFM methods due to the trade-off between object-level coherence (favored by GOLD) and sharp boundary precision (favored by SAM).

• Cross-Scene Generalization:

  • Visualizations demonstrate that the same object in different scenes is assigned the same ID using the global codebook. In contrast, VFM-based methods assign different IDs to the same object across scenes.
  • The method can discover and identify newly appearing objects in a scene without manual re-prompting, a capability where memory-based tracking (like SAM2+DEVA) fails.

• Comparison with NeRF-based OCL (uOCF):

  • Compared to uOCF (a NeRF-based OCL method), the proposed 3DGS approach achieves superior reconstruction quality (higher PSNR/SSIM) and significantly faster rendering speeds (~106 FPS vs. ~6 FPS).
  • Crucially, the 3DGS reconstruction remains intact even if object discovery is imperfect, whereas NeRF-based methods often suffer from missing geometry if objects are not discovered.

5. Significance

This work represents a paradigm shift in 3D scene understanding by moving away from appearance-driven, scene-specific supervision toward intrinsic, scene-agnostic object representation.

• For Robotics and Embodied AI: The ability to recognize and interact with objects based on consistent, learned identities across different environments is critical for generalization in real-world robotics.
• Efficiency: By leveraging 3DGS, the method offers real-time rendering capabilities, making it viable for interactive applications.
• Scalability: The elimination of complex mask alignment pipelines reduces the engineering overhead, making the approach more robust and easier to deploy in diverse scenarios.

The paper concludes that while current limitations exist regarding dataset availability and computational costs for training the global codebook, the proposed framework establishes a strong foundation for universal, object-centric 3D scene representation.