SpaCeFormer: Fast Proposal-Free Open-Vocabulary 3D Instance Segmentation

1. Problem Statement

Open-vocabulary 3D instance segmentation aims to identify and segment arbitrary objects in 3D scenes based on natural language descriptions, a capability critical for robotics, AR/VR, and embodied AI. However, existing approaches face two primary bottlenecks:

• Multi-stage Pipelines: Current state-of-the-art methods often rely on aggregating 2D foundation model outputs (e.g., SAM, CLIP) into 3D. These pipelines are computationally expensive (hundreds of seconds per scene), suffer from compounding errors across stages, and require external region proposals.
• Pseudo-labeled End-to-End Approaches: Methods attempting to train end-to-end networks often rely on single-view pseudo-labeling. This results in fragmented 3D masks (due to occlusion and incomplete 2D-to-3D lifting) and inconsistent captions across viewpoints. Furthermore, existing 3D backbones (like Point Transformer v3) often use space-filling curves (e.g., Morton codes) that scatter spatially adjacent points across attention windows, degrading the spatial coherence needed for sharp instance boundaries.

2. Methodology

The authors propose SpaCeFormer, a unified solution comprising a large-scale dataset and a novel network architecture designed for proposal-free, open-vocabulary 3D instance segmentation.

A. Dataset: SpaCeFormer-3M

To address the lack of high-quality training data, the authors constructed SpaCeFormer-3M, the largest open-vocabulary 3D instance segmentation dataset to date.

• Scale: Contains 604K instance masks and 3.0 million multi-view-consistent captions across 7,400 scenes (derived from ScanNet, ScanNet++, ARKitScenes, and Matterport3D).
• Pipeline:
  1. Multi-view Mask Clustering: Instead of naive 2D-to-3D lifting, they use training-free multi-view clustering (based on MaskClustering) to aggregate partial 2D masks into complete, geometry-consistent 3D instances. This achieves 54.3% mask recall at IoU>0.5, a 21× improvement over prior single-view pipelines (2.5%).
  2. Multi-view Captioning: They employ a structured multi-view prompting strategy for Vision-Language Models (VLMs). By presenting multiple views of an object simultaneously, the VLM generates descriptions focused on intrinsic properties (shape, material) rather than view-specific artifacts, ensuring caption consistency.

B. Architecture: SpaCeFormer (Space-Curve Transformer)

The network is an end-to-end architecture consisting of a Space-Curve Attention Backbone and a Proposal-Free Decoder.

• Space-Curve Attention Backbone:

  • Hybrid Strategy: Combines Spatial Window Attention with Morton Curve Serialization.
    • Spatial Windows: Preserve local geometric neighborhoods (critical for instance boundaries) by processing voxels within fixed 3D geometric extents.
    • Morton Curves: Provide structured diversity and long-range context at coarser scales.
  • Benefit: This combination reduces the average intra-window spatial distance by 28.6% compared to Morton-only attention, ensuring features are spatially coherent for precise boundary prediction.
  • 3D RoPE: Utilizes 3D Rotary Positional Embeddings (RoPE) to encode relative spatial relationships directly into the attention mechanism, allowing the model to reason about 3D geometry.

• Proposal-Free Decoder:

  • Mechanism: Unlike methods requiring external region proposals (e.g., Mask3D), this decoder predicts instance masks directly from a fixed set of learned queries.
  • Process: It iteratively refines queries via cross-attention with scene features and self-attention among queries.
  • RoPE Integration: Applies 3D RoPE to keys in the cross-attention to preserve absolute 3D coordinates, enabling the model to distinguish between multiple instances of the same category in different locations.
  • Output: Directly predicts instance masks, foreground/background logits, and CLIP-aligned embeddings for open-vocabulary classification.

3. Key Contributions

1. SpaCeFormer-3M Dataset: A massive, high-quality dataset with 21× higher mask recall than previous pseudo-label datasets, constructed via multi-view mask clustering and consistent captioning.
2. Space-Curve Attention: A novel attention mechanism that strategically combines spatial window attention and Morton serialization to balance local geometric coherence with global diversity.
3. Proposal-Free Architecture: An end-to-end decoder that predicts masks directly from learned queries without relying on external region proposals or multi-stage heuristics.
4. State-of-the-Art Performance: Achieves interactive inference speeds while surpassing prior methods in accuracy.

4. Experimental Results

The model was evaluated on Replica, ScanNet++, and ScanNet200 benchmarks.

• Speed: SpaCeFormer processes a scene in 0.14 seconds, making it 2–3 orders of magnitude faster than multi-stage 2D+3D pipelines (which take hundreds of seconds). It is the only method entering the "interactive" (<1s) band.
• Replica (Zero-Shot): Achieves 24.1 mAP, surpassing all prior open-vocabulary methods. It is comparable in accuracy to SOLE (24.7 mAP) but is ~9× faster and does not require ground-truth mask supervision during training.
• ScanNet++: Achieves 22.9 mAP, outperforming OpenTrack3D (20.6 mAP) despite OpenTrack3D using multi-view 2D inputs and taking ~320 seconds per scene.
• ScanNet200: Achieves 11.1 mAP in the strict "proposal-free, 3D-only, no GT" setting. This is a 2.8× improvement over the next-best method in this regime (Mosaic3D+Decoder at 3.9 mAP).
• Mask Quality: Even without semantic labels, SpaCeFormer achieves 22.5 AP (class-agnostic), outperforming multi-stage 3D+2D methods like MaskClustering (19.2 AP).

5. Significance

SpaCeFormer represents a paradigm shift in 3D instance segmentation by demonstrating that end-to-end, proposal-free learning is viable and superior to multi-stage pipelines when supported by high-quality, geometry-consistent data.

• Efficiency: It bridges the gap between accuracy and latency, enabling real-time open-vocabulary understanding for robotics and AR/VR applications.
• Data Quality: It highlights that the bottleneck in open-vocabulary 3D learning is often data quality (fragmentation/inconsistency) rather than model capacity, providing a blueprint for future dataset construction.
• Architecture: The Space-Curve attention mechanism offers a new design principle for 3D transformers, proving that preserving local spatial coherence is essential for instance-level tasks.

In summary, SpaCeFormer delivers a fast, accurate, and fully 3D-native solution for open-vocabulary instance segmentation, eliminating the need for slow, error-prone 2D aggregation pipelines.