Technical Summary: Deformable Gaussian Occupancy (DeGO)

Problem Statement

Dynamic 3D occupancy prediction is critical for autonomous driving, particularly for reasoning about human-centric, nonrigid agents. However, existing weakly supervised frameworks face two primary limitations:

1. Rigid Motion Assumptions: Most current methods rely on simple frame-to-frame offsets, assuming rigid-body motion. This prevents them from capturing fine-grained nonrigid deformations characteristic of human agents (e.g., pedestrians, cyclists).
2. Capacity Imbalance: Methods distributing Gaussian primitives uniformly across the 3D volume often allocate excessive capacity to static background regions (roads, walls), leaving insufficient resolution for small, safety-critical human instances.
3. Temporal Inconsistency: Prior voxel or Gaussian approaches struggle to maintain temporal coherence in dynamic environments where rigid and nonrigid motions coexist, leading to unstable geometry updates.

Methodology

The authors propose DeGO (Deformable Gaussian Occupancy), a framework that unifies decoupled Gaussian deformation with factorized 4D foundation-model distillation. The method operates under weak supervision, utilizing pseudo-labels from Grounded-SAM and Metric3D rather than dense voxel annotations.

1. Decoupled Gaussian Deformation (DGD)

Instead of treating all classes uniformly, DeGO enables each Gaussian primitive to adaptively model its motion behavior.

• Soft Rigidity Mask: Each Gaussian is augmented with a learnable rigid-body mask ($m_i \in [0, 1]$). This mask implicitly controls whether a Gaussian undergoes only rigid motion or a combination of rigid offsets and nonrigid deformation.
• Hybrid Motion Updates:
  • Rigid Regions ($m_i \approx 0$): Gaussians are updated solely via positional offsets ($\Delta \mu$).
  • Nonrigid Regions ($m_i \approx 1$): Gaussians undergo both positional offsets and local deformations affecting rotation ($r$), scale ($s$), and opacity ($\alpha$).
• Regularization: A binary regularization loss encourages the mask to converge toward near-binary values, ensuring distinct separation between rigid and nonrigid behaviors.

2. Factorized Feature Distillation (FFD)

To enhance temporal consistency and multi-view alignment, DeGO distills knowledge from a 4D foundation model, VGGT.

• 4D Teacher Features: Unlike previous methods that distill per-frame 2D features, FFD extracts spatiotemporal representations from VGGT. The teacher model processes multi-view sequences using interleaved cross-camera (spatial) and cross-frame (temporal) attention mechanisms.
• Distillation Process: The student model (DeGO) aligns its rendered Gaussian features with the teacher's 4D features. This transfers cross-camera and cross-frame reasoning capabilities, providing stable guidance for the Gaussian field without requiring explicit 3D labels.
• Architecture: The distillation targets the Gaussian Transformer, aligning the 3D Gaussian features with the foundation model's spatiotemporal tokens before the deformation heads predict motion updates.

3. Optimization and Inference

• Training: The model is trained on short sequences of multi-view images. The total loss combines segmentation loss, depth regression loss, feature distillation loss, and deformation regularization.
• Inference: During deployment, the system operates in a single-frame, feed-forward manner. The temporal deformation module is used during training to learn motion-aware priors but does not require temporal input at inference time. The output is a single set of 3D Gaussians voxelized for downstream occupancy prediction.

Key Contributions

1. Unified Deformable Framework: A novel occupancy framework where each Gaussian adaptively models rigid and nonrigid motion, guided by 4D foundation models to improve both human-centric and overall scene understanding.
2. Human-Centric Decoupled Deformation: A mechanism that selectively allocates deformation capacity, enabling accurate modeling of nonrigid agents while preserving geometric stability for rigid structures.
3. Factorized 4D Distillation: A strategy that transfers cross-camera and cross-frame knowledge from foundation models (VGGT) to the Gaussian field, creating foundation-aligned dynamic scene representations.

Experimental Results

Experiments were conducted on the Occ3D-NuScenes benchmark (40k frames, 6 camera views).

• Overall Performance: DeGO achieves state-of-the-art performance under weak supervision, improving the overall mIoU by 10.9% and IoU by 12.4% compared to the previous best weakly supervised method (GaussianFlow).
• Human-Centric Metrics: The method demonstrates significant gains in modeling nonrigid agents, achieving a 13.5% improvement in the proposed Human-centric mIoU (HCM) and a 7.8% gain in Instance mIoU (InsM).
• Ablation Insights:
  • The deformation module alone provided a 43.4% improvement over the baseline.
  • Using VGGT as the teacher yielded higher performance than DINOv2, attributed to VGGT's pre-training on multi-view and temporal data.
  • Distilling both spatial (cross-camera) and temporal (cross-frame) attention features provided complementary benefits, with the combined approach yielding the best results.
  • The rigid-body mask was identified as the most critical component for distinguishing motion types.

Significance

The paper claims that DeGO addresses the fundamental limitations of current occupancy prediction methods by explicitly separating rigid and nonrigid dynamics. By combining deformation-aware modeling with foundation-guided distillation, the approach achieves robust temporal consistency and superior reconstruction of safety-critical human instances without relying on expensive 3D voxel labels. The results highlight the effectiveness of using 4D foundation models to guide weakly supervised learning in dynamic scenes.