EnerGS: Energy-Based Gaussian Splatting with Partial Geometric Priors

1. Problem Statement

3D Gaussian Splatting (3DGS) has revolutionized novel view synthesis by offering real-time rendering and high photorealistic quality. However, in large-scale, unbounded outdoor scenarios (e.g., autonomous driving), 3DGS faces significant challenges:

• Ill-posed Optimization: In sparse-view regimes, the optimization problem is ill-posed. Without sufficient constraints, the system tends to overfit training views, creating geometrically invalid artifacts known as "floaters" (Gaussian primitives floating in empty space) or near-camera distortions.
• Limitations of Existing Geometric Priors: Recent methods incorporate geometric priors (e.g., LiDAR point clouds) to constrain optimization. However, LiDAR data in outdoor scenes is often spatially incomplete and uneven (e.g., limited vertical Field-of-View, sparsity).
  • Treating LiDAR as a uniform constraint fails because geometric unobservability does not imply visual occlusion. For example, rooftops or distant backgrounds may be invisible to LiDAR but clearly visible to cameras.
  • Applying rigid depth constraints globally can suppress valid structures in unobserved regions.
  • Conversely, lacking strict exclusion mechanisms in verified free space allows artifacts to accumulate.

Core Challenge: How to leverage partial, sparse, and noisy geometric priors (LiDAR) to guide 3DGS optimization without suppressing valid geometry in unobserved regions or allowing artifacts in known free space.

---

2. Methodology: EnerGS

The authors propose EnerGS, a framework that reformulates 3DGS optimization as inference within a continuous geometric energy field. Instead of hard constraints, EnerGS uses a spatially adaptive energy potential to provide soft guidance.

A. Spatial Partitioning & Energy Field

The 3D space $\Omega$ is partitioned into three distinct regimes based on sensor observability:

1. Occupied Space ($\Omega_{occ}$): Regions with LiDAR points.
   • Mechanism: Uses a Welsch M-estimator as a volumetric attractor.
   • Function: Pulls Gaussian primitives toward the nearest LiDAR point. It is robust to outliers (bounded influence) unlike standard $L_2$ loss, preventing distant noise from distorting the geometry.
2. Free Space ($\Omega_{free}$): Regions where LiDAR rays have passed without hitting anything (verified empty).
   • Mechanism: Uses a Boltzmann barrier (implemented via a Softplus function).
   • Function: Acts as a strictly monotonic repulsor. It imposes a linear energy cost for penetration depth, pushing primitives out of free space to eliminate floaters.
3. Unknown Space ($\Omega_{unk}$): Regions with no LiDAR coverage (e.g., sky, rooftops, far-field).
   • Mechanism: Uses a weak prior (Welsch estimator with high variance).
   • Function: The energy gradient is near-zero. This allows the optimization to remain compliant, letting photometric cues (camera images) drive the reconstruction of geometry in these "blind spots" without being constrained by incorrect geometric assumptions.

B. Decoupled Optimization Strategy

A key innovation is the decoupling of geometric and appearance updates to resolve gradient conflicts:

• Standard Approach: Jointly minimizes Photometric Loss ($L_{photo}$) and Geometric Loss ($L_{geom}$). This often leads to conflicts where noisy photometric gradients override geometric signals, trapping the optimizer in local minima (floaters).
• EnerGS Approach:
  • Position Update ($\mu$): Updated solely by the gradient of the geometric energy field ($\nabla E_{geom}$). This guarantees monotonic descent on the geometric landscape, ensuring primitives are physically valid (e.g., expelled from free space).
  • Appearance/Shape Update ($\Sigma, \alpha, c$): Updated solely by the photometric loss ($L_{photo}$).
  • Result: This separation ensures that geometry is stabilized by the energy field while appearance is optimized for visual fidelity, preventing the "memorization" of training views.

C. Discrete Pruning

As a secondary safety mechanism, the method employs discrete hard pruning. Periodically, any Gaussian primitive found deep within verified free space (beyond a margin threshold) is explicitly removed to prevent the accumulation of artifacts that the continuous field might miss.

---

3. Key Contributions

1. Energy-Based Framework: Introduced a differentiable geometric energy field that unifies uncertainty-aware occupancy attraction (Welsch M-estimator) and free space exclusion (Boltzmann barrier) into a single potential.
2. Gradient Decoupling: Proposed a decoupled optimization strategy that theoretically guarantees the elimination of stable degenerate solutions (floaters) in trusted free space while preserving flexibility in unobserved regions.
3. Theoretical Analysis: Proved that:
   • Degenerate solutions cannot persist in trusted free space (Theorem 1).
   • The geometric update field is Lipschitz continuous, ensuring training stability (Theorem 2).
   • Geometric constraints naturally vanish in unknown regions, allowing photometric densification (Proposition 3).
4. State-of-the-Art Performance: Demonstrated superior results on large-scale autonomous driving datasets (KITTI, Waymo), significantly reducing geometric violations while maintaining high rendering quality.

---

4. Experimental Results

The method was evaluated on KITTI and Waymo Open Dataset against baselines like vanilla 3DGS, 2DGS, GeoGaussian, and Mip-Splatting.

• Quantitative Metrics:
  • Geometric Quality: EnerGS achieved the lowest "Leak" scores (fewer primitives in free space) and highest "Margin" scores (better separation between occupied and free space). It also showed higher OccCov (better coverage of occupied space).
  • Photometric Quality: Achieved the highest PSNR and competitive SSIM on both datasets, outperforming methods that struggle with sparse views.
  • Efficiency: Maintained a comparable number of Gaussian primitives to baselines, indicating a more compact and physically consistent representation.
• Qualitative Results:
  • In regions beyond LiDAR coverage (e.g., rooftops, sky, distant trees), EnerGS successfully reconstructed fine details driven by camera images, whereas baselines either failed to reconstruct them or produced floaters.
  • In verified free space, EnerGS completely eliminated the "floating" artifacts common in other methods.
• Ablation Studies: Confirmed that removing any component (e.g., the free space barrier, the unknown weak prior, or the decoupled optimization) led to consistent degradation in both geometric stability and rendering quality.

---

5. Significance

EnerGS addresses a critical bottleneck in applying 3DGS to real-world, large-scale robotics and autonomous driving scenarios.

• Robustness to Sensor Limitations: It provides a principled way to handle the inherent sparsity and incompleteness of LiDAR sensors without relying on perfect geometric supervision.
• Bridging the Gap: By distinguishing between "verified free space," "verified occupied space," and "unknown space," it bridges the gap between deterministic sensor data and ambiguous visual data.
• Generalizability: The energy-based formulation is modular and can be integrated into other Gaussian Splatting pipelines (as demonstrated with SplatAD and StreetGS), making it a versatile tool for improving 3D reconstruction in unbounded environments.

In summary, EnerGS transforms the optimization of 3D Gaussians from a purely photometric task into a physics-informed process, ensuring that the resulting 3D scene is not only visually realistic but also geometrically plausible and free of artifacts.