Technical Summary: Gaussian Sculpting

Problem Statement

While 3D Gaussian Splatting (3DGS) has achieved impressive real-time novel view synthesis, it struggles to recover accurate, watertight surfaces, particularly under limited viewpoints. The inherent irregularity of Gaussian primitives leads to geometric errors, floating artifacts, and incomplete structures that are difficult to correct manually. Existing approaches typically treat surface reconstruction as a post-processing step (e.g., applying Marching Cubes to optimized Gaussians), which decouples geometry from appearance optimization. This decoupling results in surfaces that are highly sensitive to noise and fail to align faithfully with the underlying geometry. Furthermore, recent field-optimized Gaussian methods often suffer from discretization errors, such as floating surfaces and sliver triangles, because the irregular Gaussian distribution cannot naturally align with a triangle mesh.

Methodology

The authors propose Gaussian Sculpting, a fully differentiable, end-to-end framework that optimizes a Signed Distance Field (SDF) rather than Gaussian primitives directly. The core insight is to anchor Gaussians onto an evolving differentiable surface, allowing them to guide SDF optimization rather than serving as the sole source of geometry.

Key Components:

1. Implicit Field Representation & Differentiable Extraction:

   • The SDF is represented implicitly using a Multi-Layer Perceptron (MLP), providing continuous geometry and better resolution control compared to discrete voxel-based SDFs.
   • The framework extends FlexiCubes to support multi-resolution grids and SDF inputs, enabling differentiable iso-surface extraction with improved topological consistency.

2. Surface-Constrained Gaussian Model:

   • Instead of treating Gaussians as independent variables, they are instantiated on the extracted mesh.
   • Parameterization: Gaussian centers are defined as convex combinations of triangle vertices. Their orientation is tied to face normals and edge directions, and their scale is initialized based on triangle edge lengths.
   • Constraints: To ensure consistency between the Gaussian rendering and the underlying surface, three specific constraints are imposed:
     • Scale Constraint: Ensures Gaussians remain within the face-aligned minimum enclosing ellipse (MEE) to prevent excessive overlap.
     • Distribution Constraint: Includes boundary avoidance, distance regularization, and coverage loss to prevent degenerate placements and ensure stable coverage over triangular faces.
     • Opacity Constraint: Fixes Gaussian opacity close to 1 (via inverse sigmoid) to prevent low-opacity Gaussians from hiding erroneous geometry, ensuring stable rendering supervision.

3. Bi-Level Optimization Strategy:

   • To stabilize training and prevent unreliable gradients from contaminating geometry optimization, a bi-level loop is employed:
     • Inner Loop: Optimizes Gaussian parameters (position, scale, color) conditioned on a fixed surface geometry. A detached copy of the Gaussians is used to prevent gradients from flowing back to the SDF during this phase.
     • Outer Loop: Updates the SDF and surface geometry using the photometric loss computed from the optimized Gaussians.
   • This gradient isolation ensures that rendering supervision guides the geometry without being corrupted by the immediate state of the Gaussians.

4. Resolution Control:

   • An octree-like adaptive subdivision scheme is used. Coarse voxels are used in empty regions, while surface-intersecting voxels are subdivided into finer grids. This reduces memory consumption while preserving fine details where needed.

Key Contributions

• End-to-End Framework: The proposal of Gaussian Sculpting, which unifies SDF optimization and constrained Gaussian rendering into a single differentiable pipeline.
• Geometry-Aware Gaussians: The identification of the inconsistency between standard Gaussian rendering and surface geometry, addressed by introducing strict geometric and opacity constraints to align Gaussians with the underlying mesh.
• Stable Optimization: The development of a bi-level training strategy with gradient isolation and progressive subdivision, which stabilizes geometry refinement and prevents artifact propagation.
• Superior Reconstruction: The demonstration of cleaner, more complete, and topologically regular meshes compared to prior NeRF-based and Gaussian-based methods, effectively reducing floating artifacts and recovering missing regions.

Experimental Results

The method was evaluated on the NeRF Synthetic and OmniObject3D datasets against a wide range of baselines, including NeRF variants (NeuS, VolSDF), 3DGS-based methods (2DGS, PGSR, GOF), and joint approaches (GSDF, 3DGSR).

• Quantitative Performance: Gaussian Sculpting achieved the lowest Chamfer Distance (CD) across the majority of scenes in both datasets. For instance, on OmniObject3D, it achieved a mean CD of 9.09 (x10⁻³), outperforming the next best method (GOF at 14.91) and significantly reducing errors in challenging objects like walnuts and pumpkins where other methods failed or produced large errors.
• Mesh Quality: The method produced meshes with significantly fewer sliver triangles and more regular shapes compared to baselines. Unlike methods that produce cluttered or irregular triangle distributions, Gaussian Sculpting adhered more closely to Delaunay triangulation principles.
• Robustness: The approach demonstrated robustness in recovering geometric continuity in regions with missing viewpoints, a common failure point for TSDF fusion and other Gaussian-based methods.
• Ablation Studies: Experiments confirmed that the opacity constraint is critical for preventing floating artifacts, while scale and distribution constraints are essential for accurate surface alignment and mesh quality. The use of neural implicit SDFs and gradient isolation was also shown to be superior to discrete SDFs and non-isolated training.

Significance and Claims

The paper claims that Gaussian Sculpting addresses the fundamental challenge of obtaining watertight, clean, and continuously optimizable meshes from limited-view RGB images. By reformulating surface reconstruction as an SDF optimization problem guided by constrained Gaussians, the method overcomes the limitations of post-processing extraction and the geometric inconsistencies of direct Gaussian optimization.

The authors position their work as a solution to the "floating artifacts" and "geometric incompleteness" prevalent in current state-of-the-art methods. They assert that their framework achieves strong reconstruction quality even at relatively low resolutions and produces meshes that are more suitable for downstream tasks (such as game asset modeling and AR/VR) which require editable and structured geometric models.

Limitations Acknowledged:
The authors note that the bi-level optimization leads to longer reconstruction times (6–18 hours per scene) and high GPU memory usage, currently restricting the framework to object-level scenes. Additionally, rendering supervision remains challenging for highly reflective materials and fine-scale textures where geometric cues are ambiguous. Future work is proposed to reduce runtime overhead and improve handling of reflective surfaces.