Technical Summary: GhostPoint

Problem Statement

The paper addresses a systematic transfer gap in Self-Supervised Learning (SSL) for LiDAR point clouds. While recent SSL methods (e.g., Sonata, DOS, Occ-MAE) effectively pretrain representations for per-point tasks like semantic and panoptic segmentation, their transfer to 3D object detection remains significantly weaker.

The authors identify the root cause as a representational mismatch:

1. SSL Objectives: Current SSL methods define learning signals exclusively on measured LiDAR returns (visible surfaces). They treat voxels without point returns as empty or ignore them, failing to model occluded or unobserved regions.
2. Detection Requirements: 3D object detection requires inferring the full geometric extent, position, and orientation of objects, which often involves reasoning about occluded and unobserved space.
3. The Consequence: Pretrained encoders learn features biased toward visible surfaces. When applied to detection, the predicted instance centroids are skewed toward observed points rather than the true geometric center of the object, leading to poor localization and extent estimation, particularly under heavy occlusion or sparse scans.

Methodology: GhostPoint

GhostPoint is an SSL framework designed to bridge this gap by explicitly modeling latent features in unobserved local neighborhoods around discovered instances. The framework operates on two levels: Encoder-level (standard self-distillation) and Predictor-level (hallucination of unobserved structure).

1. Instance-Centric Neighborhood Sampling

Instead of uniform sampling in free space, GhostPoint focuses on regions adjacent to objects where occlusions are most likely to occur.

• Pseudo-Instance Discovery: A teacher encoder processes unmasked LiDAR returns to generate pseudo-instances using Softmap (semantic) and center-offset heads.
• Visibility Filtering: Instances fully masked in the student view are discarded.
• Instance Voxel Dilation: For remaining instances, the binary occupancy grid is dilated using a 3D kernel ($k_s > 1$). This creates a neighborhood voxel set $\mathcal{Q}$ that includes:
  • $Q_{vis}$: Visible occupied voxels.
  • $Q_{mask}$: Masked occupied voxels (pseudo-occlusion).
  • $Q_{unobs}$: Unobserved voxels (no-return regions).

2. Token Initialization and Refinement

The student predictor must generate features for voxels in $\mathcal{Q} \setminus Q_{vis}$ (masked and unobserved) where no encoder features exist.

• Initialization: Tokens for these new voxels are initialized via distance-weighted KNN interpolation from the nearest $k$ visible occupied neighbors.
• Prediction: A lightweight predictor ($G_\theta$) processes the neighborhood tokens to aggregate context from visible voxels into the new tokens.

3. Asymmetric Teacher-Student Distillation

GhostPoint introduces a predictor-level supervision scheme that differs for observed vs. unobserved voxels:

• Observed Voxels ($Q_{vis} \cup Q_{mask}$): The student predictor is supervised by the teacher encoder features. This aligns with standard self-distillation.
• Unobserved Voxels ($Q_{unobs}$): The student predictor is supervised by the teacher predictor outputs. Since the teacher predictor is an Exponential Moving Average (EMA) of the student and conditioned on the full unmasked scene, it "hallucinates" plausible semantic and geometric representations for these empty regions.

4. Training Objectives

The total loss combines encoder-level and predictor-level distillation:
$$\mathcal{L} = \mathcal{L}_{\text{sem,vis}} + \lambda \mathcal{L}_{\text{geo,vis}} + \mathcal{L}_{\text{sem},\neg \text{vis}} + \lambda \mathcal{L}_{\text{geo},\neg \text{vis}}$$

• $\mathcal{L}_{\text{vis}}$: Standard Softmap and offset losses on visible voxels.
• $\mathcal{L}_{\neg \text{vis}}$: KL divergence for semantic distributions and offset loss for geometric predictions on non-visible voxels (masked and unobserved).

Key Contributions

1. Analysis of the Transfer Gap: The paper characterizes the limitation of surface-aligned SSL objectives for 3D detection, attributing the performance drop to the inability of current methods to reason about unobserved object geometry.
2. GhostPoint Framework: A novel SSL approach that uses instance voxel dilation to generate neighborhood queries and a predictor-level distillation scheme to hallucinate latent features in unobserved regions.
3. Empirical Validation: Demonstration that explicitly modeling unobserved structure improves downstream 3D detection performance, particularly in sparse scanning conditions and low-label regimes, without sacrificing segmentation capabilities.

Experimental Results

The method was evaluated on nuScenes and Waymo Open Dataset benchmarks.

• Performance Gains:

  • Probing (Frozen Backbone): GhostPoint outperforms the second-best SSL method (PointINS) by 2.8 mAP and 1.7 NDS on nuScenes, and 2.5 mAP on Waymo.
  • Full Fine-tuning: GhostPoint achieves 67.5 mAP and 71.2 NDS on nuScenes, surpassing the fully supervised baseline by 3.7 mAP.
  • Qualitative Improvements: Visualizations show GhostPoint produces predictions with better center localization and spatial extent alignment, especially for partially occluded objects.

• Label Efficiency:

  • With only 0.1% of labels, GhostPoint achieves 36.9 mAP on nuScenes, outperforming the second-best SSL method by 2.1 mAP.
  • At 10% labels, it matches or exceeds the performance of fully supervised training in certain metrics.

• Ablation Studies:

  • Neighborhood Sampling: Extending supervision to occluded regions via dilation alone yields improvements.
  • Predictor: The addition of the predictor and warmup schedule provides substantial gains, confirming the need for active context propagation.
  • Hallucination vs. Regularization: Experiments confirm that gains come specifically from informative hallucination targets in unobserved regions, not merely from increased model capacity or regularization.
  • Architecture Transfer: Pretraining a Transformer (PTv3) and distilling to a sparse CNN (CenterPoint) yields better results than training the CNN directly, suggesting broad applicability.

• Task Transferability: GhostPoint maintains strong performance on semantic and panoptic segmentation (matching PointINS on mIoU), proving that modeling unobserved structure does not degrade per-point feature quality.

Significance and Claims

The paper claims that GhostPoint effectively closes the transfer gap between SSL and 3D object detection by shifting the learning objective from "visible surfaces" to "full object geometry." By hallucinating latent features in unobserved neighborhoods, the framework encourages the encoder to internalize object structure beyond measured returns.

The authors position this work as a scalable solution for autonomous driving, where unlabeled sensor logs are abundant but dense 3D annotations are costly. They note that while GhostPoint significantly narrows the gap to supervised training, it still trails fully supervised performance under probing conditions and has not yet been evaluated on other sparse modalities like radar.