ClickSeg3D: Few-Click Interactive Segmentation via Semantic Embeddings

Technical Summary: ClickSeg3D

Problem Statement
Interactive 3D segmentation aims to generate efficient labels by leveraging user-provided clicks to progressively refine predictions, a capability critical for scenarios where fully supervised labels are costly or generalization to unseen classes is required. Existing 3D interactive methods face significant limitations: most operate sequentially, predicting only one object per iteration with binary masks, which results in slow inference for multi-object scenes. Furthermore, recent approaches often rely on 2D foundation models (e.g., SAM) and require accurate camera alignment to bridge the 2D-3D gap. These methods are sensitive to sparse viewpoints, occlusion, and inconsistencies between 2D masks and 3D labels. Additionally, iterative pipelines that require repeated model updates after every corrective click are computationally expensive and limit scalability, particularly for unseen datasets.

Methodology
The authors propose ClickSeg3D, a novel interactive segmentation framework that operates directly on sparse, randomly downsampled 3D points and processes multiple object clicks in a single forward pass. The architecture consists of three primary components:

1. Point Transformer-Based Encoder: The model utilizes a shared Point Transformer v3 (PTv3) encoder to extract rich features from both the scene points and the user-provided click points. This encoder generates multi-scale scene features and initializes query features based on the nearest voxel to the click location.
2. Hierarchical Mask Decoder with Conditional Query Adaptor: Unlike prior iterative pipelines, the decoder processes all click queries simultaneously. It employs a multi-stage architecture where a transformer block refines features through cross-attention (click-to-scene) and self-attention (click-to-click). Crucially, the framework introduces a Conditional Query Adaptor that refines sparse click features using learnable semantic and spatial embeddings.
   • Semantic Prototype Embeddings: Inspired by open-vocabulary segmentation, the model uses learnable semantic prototypes to condition query features, enhancing attention and robustness to unseen categories.
   • Multi-Level Crop-and-Merge: The decoder progressively refines instance masks from coarse to fine granularity. It utilizes a mask head for spatial prediction and a class head for semantic classification, fusing spatial embeddings (derived from mask pooling) and semantic embeddings (selected from prototypes) to update query features via a lightweight MLP.
3. Training Strategy: The authors introduce a Sparse Interaction-Driven Training Strategy. Instead of relying on iterative human feedback during training, they simulate realistic user interactions by using Farthest Point Sampling (FPS) to generate simulated click points on downsampled scenes. This allows the model to learn a "one-click-per-instance" multi-object prediction problem in a single forward pass, where the positive click of one instance implicitly serves as negative supervision for others.

Key Contributions

• Decoupled Single-Pass Architecture: The framework explicitly disentangles point feature extraction from multi-stage mask refinement, enabling efficient, fine-grained multi-object segmentation in a single forward pass without iterative re-inference.
• Sparse Interaction-Driven Training: A training paradigm based on sparsely and randomly downsampled points simulates user interactions, improving robustness to varying click numbers and spatial distributions while maintaining training efficiency.
• Multi-Stage Crop-and-Merge Decoder: A progressive decoder that leverages learnable semantic prototype embeddings to condition and update query features. This enhances mask precision and generalization to unseen datasets without increasing computational cost.

Experimental Results
Extensive experiments were conducted on indoor (ScanNet40, S3DIS) and outdoor (KITTI360) datasets.

• Performance: The model improves the mean Intersection-over-Union (mIoU) by over 20% compared to strong baselines (e.g., InterObject3D, Agile3D, SAM2Point) in a one-click-per-instance setting. On ScanNet40, it achieves 71.6% mIoU with a single click, compared to 65.2% for SAM2Point and 63.3% for Agile3D.
• Generalization: Under cross-dataset evaluation (e.g., training on ScanNet40 and testing on S3DIS or KITTI360), the method achieves 8–10% gains over baselines, demonstrating superior robustness to domain shifts.
• Efficiency: The single-pass design significantly reduces computational overhead. The model achieves competitive inference latency (0.06s) and a reasonable memory footprint (128.5 MB), making it suitable for real-time applications.
• Ablation Studies: Removing components such as the semantic embeddings, spatial embeddings, or the query adaptor leads to significant performance drops, validating the necessity of the proposed architecture.

Significance and Claims
The paper claims that ClickSeg3D provides a generalizable and efficient solution for interactive 3D instance segmentation. By eliminating the need for iterative refinement and explicit camera alignment, the method addresses the scalability and efficiency bottlenecks of current approaches. The authors assert that their approach is particularly suitable for real-time applications such as robotic manipulation, navigation, and rapid 3D semantic annotation. The work demonstrates that modeling inter-instance relationships and refining spatial/semantic predictions through embeddings in a single pass can outperform methods relying on 2D foundation models or sequential inference.

Limitations
The authors acknowledge that the current framework is limited to point-cloud-based click queries and does not yet exploit richer cross-modal cues, such as text prompts or 2D mask guidance from foundation models. Additionally, user interaction is still required to initiate segmentation, and the model does not explicitly reason about fine-grained part hierarchies. Future work is proposed to extend the framework to multi-modal interactive segmentation.