Bridging 3D Gaussians and Semantic Occupancy for Comprehensive Open-Vocabulary Scene Understanding from Unposed Images

Technical Summary: COVScene

Problem Statement
The paper addresses the challenge of achieving comprehensive 3D scene understanding from sparse, unposed images without relying on external camera calibration or Structure-from-Motion (SfM) pipelines. While recent feed-forward Gaussian methods (e.g., LSM, Uni3R) have advanced pose-free reconstruction and semantic rendering, they remain largely surface-centric. These methods optimize Gaussian primitives primarily through image-space rendering losses. Consequently, in unobserved or weakly observed regions, the learned primitives lack explicit constraints on free versus occupied space. This leads to physically implausible artifacts such as "floaters," hollow structures, and ambiguous density distributions, even when the rendered RGB or semantic images appear plausible. Existing approaches often treat volumetric occupancy as an auxiliary branch or a post-hoc conversion, failing to provide volumetric feedback during the training process to correct the underlying Gaussian representation.

Methodology
The authors propose COVScene, a pose-free semantic Gaussian framework that couples renderable Gaussian primitives with a dense semantic occupancy field through differentiable volumetric lifting. The core innovation is performing this lifting within the training computation graph rather than as a detached post-processing step.

1. Architecture:

   • Semantic-Aware Geometry Transformer: The model uses a shared encoder initialized from a pretrained geometry foundation model (VGGT) to infer camera geometry, depth, and scene structure directly from unposed multi-view images. This avoids dependence on external SfM.
   • Multi-Task Decoding: A Dense Prediction Transformer (DPT) decoder branches into three heads: a Camera Head (predicting extrinsics/intrinsics), a Depth Head (estimating 3D centers), and a Semantic Gaussian Head (predicting opacity, rotation, scale, color, and open-vocabulary semantic features).
   • Differentiable Volumetric Lifting: Predicted semantic Gaussians are lifted into a dense semantic occupancy field $(O, F)$ during training. For each voxel, occupancy probability $O(x)$ is computed by accumulating Gaussian densities, and semantic features $F(x)$ are aggregated via density-weighted averaging. This process is fully differentiable, allowing gradients from volumetric losses to update the Gaussian opacity, geometry, and semantic features directly.

2. Optimization Strategy:
   The framework employs a multi-objective loss function combining:

   • Photometric and Semantic Supervision: Standard rendering losses (L1, LPIPS) and feature-level alignment with frozen vision-language models (e.g., CLIP) for open-vocabulary capabilities.
   • Geometric Foundation Distillation: Distillation from a frozen geometry teacher (VGGT) to provide initial multi-view geometric scaffolds and reduce scale ambiguity.
   • Occupancy Entropy Regularization: A critical component that enforces a bimodal physical prior on the lifted occupancy field. By minimizing the entropy of the occupancy probability, the model penalizes ambiguous states ($O(x) \approx 0.5$) and encourages regions to be either clearly free ($O(x) \to 0$) or occupied ($O(x) \to 1$). This resolves geometric inconsistencies without requiring direct voxel-level supervision.

Key Contributions

• Unified Representation: COVScene is the first pose-free framework to couple renderable Gaussian primitives with a dense semantic occupancy field via differentiable lifting, enabling novel view synthesis, open-vocabulary segmentation, and semantic occupancy prediction within a single representation.
• Training-Time Volumetric Feedback: Unlike prior works that convert Gaussians to voxels only at evaluation, COVScene integrates volumetric regularization into the training loop. This allows volumetric losses to directly regularize Gaussian opacity and geometry, reducing artifacts in unobserved regions.
• Entropy Regularization: The introduction of an occupancy entropy objective enforces physically plausible free/occupied structures, effectively suppressing "floaters" and hollow artifacts common in unposed 3DGS.
• Zero-Shot Capabilities: The method supports open-vocabulary semantic querying and zero-shot 3D occupancy prediction without extensive category-level annotations.

Experimental Results
The authors evaluated COVScene on the ScanNet and ScanNet++ datasets across three tasks:

• Novel View Synthesis (NVS): COVScene maintains competitive rendering quality (PSNR, SSIM, LPIPS) compared to state-of-the-art feed-forward Gaussian baselines.
• Open-Vocabulary Segmentation (OVS): The framework demonstrates improved performance in open-vocabulary segmentation compared to self-supervised baselines and other semantic Gaussian methods.
• Semantic Occupancy Prediction: COVScene achieves stronger semantic occupancy prediction than self-supervised baselines (e.g., SelfOcc) and competitive results against fully supervised methods, despite lacking direct voxel-level supervision during training.
• Ablation Studies: Experiments confirm that simply converting semantic Gaussians to occupancy at evaluation time is insufficient. Training-time volumetric grounding is shown to be essential for reducing geometric artifacts and improving occupancy accuracy.

Significance and Claims
The paper claims that COVScene establishes a new paradigm for scalable 3D scene understanding by bridging the gap between efficient, renderable Gaussian representations and physically grounded volumetric reasoning. By treating occupancy not as an auxiliary output but as a native, differentiable constraint on the Gaussian representation, the method achieves a more physically plausible 3D reconstruction. The authors position COVScene as a significant step toward robust 3D understanding for robotics and embodied agents, where accurate reasoning about free and occupied space is critical, all while operating without the need for calibrated camera poses. The work demonstrates that volumetric regularization can be effectively integrated into feed-forward Gaussian pipelines to resolve the inherent ambiguities of sparse, unposed inputs.