MeshSplats: Mesh-Based Rendering with Gaussian Splatting Initialization

1. Problem Statement

Gaussian Splatting (GS) has emerged as a state-of-the-art technique for 3D scene reconstruction, offering real-time rendering and high-quality results by representing scenes as sets of anisotropic 3D Gaussians. However, GS relies on rasterization rather than ray tracing. This creates significant limitations:

• Lighting Effects: Rasterization struggles to handle incoherent rays required for advanced lighting effects like accurate shadows, reflections, and global illumination.
• Pipeline Incompatibility: GS models cannot be directly utilized in standard, high-fidelity rendering engines (e.g., Blender, Nvdiffrast) that rely on mesh-based representations and ray tracing.
• Editing Constraints: While some methods attempt to integrate ray tracing with GS (e.g., 3DGRT, RaySplats), they often require specialized, dedicated renderers and do not produce standard mesh formats suitable for manual editing or physics simulations.

The core challenge is to convert the flexible, high-quality GS representation into a mesh-based format that retains visual fidelity while enabling compatibility with standard ray-tracing pipelines and editing tools.

2. Methodology

The authors propose MeshSplats, a method that acts as a "wrapper" to transform pre-trained Gaussian Splatting models into disjoint mesh representations. The approach operates in two primary modes and involves a specific transformation pipeline followed by optimization.

A. Transformation Pipeline

The core mechanism converts Gaussian primitives (ellipsoids or flat ellipses) into mesh faces (triangles).

1. Initialization (GS Prior):

   • Flat Gaussians (2DGS): For Gaussians constrained to 2D manifolds, the method approximates the ellipse as a triangle fan. It computes two orthogonal vectors based on the Gaussian's scale and rotation, generates vertices along the ellipse boundary, and connects them to the center.
   • 3D Gaussians: For full 3D Gaussians, the method approximates the ellipsoid using three orthogonal triangle fans (representing the principal axes). This captures the full 3D extent but results in higher memory usage.
   • Attributes: Vertex colors and opacities are inherited directly from the source Gaussians. Opacity is scaled at the boundaries to approximate the Gaussian fall-off.

2. Mesh Prior Mode (Advanced):

   • Instead of converting Gaussians directly, this mode uses an external tool to estimate a base geometry mesh of the object.
   • MeshSplats then places a secondary layer of disjoint, colored triangles directly onto the faces of this geometry mesh.
   • Parameterization: The vertices of the new triangles are defined as convex combinations of the underlying geometry mesh vertices. This ensures the generated mesh adheres strictly to the object's shape, facilitating precise manual editing.

B. Optimization and Refinement

After the initial transformation, the resulting "mesh soup" often contains artifacts (e.g., floating triangles, lack of fine detail). The authors introduce a dedicated optimization pipeline:

• Rendering: The mesh is rendered using Nvdiffrast (a differentiable rasterizer) to compute gradients.
• Loss Function: A weighted combination of L1 loss and SSIM (Structural Similarity Index Measure) is used to minimize the difference between the rendered image and the ground truth.
• Optimization:
  • Vertex Positions: Optimized with a low learning rate to ensure stability.
  • Colors/Opacities: Optimized to refine appearance.
• Pruning: Redundant faces and vertices (those with very low opacity) are removed every 10 epochs to reduce memory usage and clean up the geometry.

3. Key Contributions

1. Novel Conversion Method: Introduction of MeshSplats, a technique that transforms GS models (both 2D and 3D Gaussians) into standard mesh formats (triangle soups) without requiring retraining from scratch.
2. Ray Tracing Compatibility: The resulting meshes are immediately compatible with standard graphics engines (Blender, Nvdiffrast), enabling the use of ray tracing for shadows, reflections, and complex lighting effects that are impossible with standard GS rasterization.
3. Dual-Mode Flexibility:
   • GS Prior Mode: Fast conversion for high-quality rendering with minimal tuning.
   • Mesh Prior Mode: Enables direct manual editing and physics simulation by anchoring the rendering elements to a structural geometry mesh.
4. Optimization Pipeline: A dedicated post-processing algorithm that refines the mesh, removes "floaters" (artifacts), and enhances geometric details, achieving photorealistic results.

4. Experimental Results

The authors evaluated MeshSplats on standard datasets: Mip-NeRF 360, Tanks and Temples, and Deep Blending.

• Quantitative Performance:

  • MeshSplats achieves comparable PSNR, SSIM, and LPIPS scores to state-of-the-art rasterization-based methods (like 3DGS and RaySplats).
  • On the Deep Blending dataset (indoor scenes with complex geometry), MeshSplats showed superior performance, outperforming several baselines and approaching the metrics of 3DGS.
  • In comparisons with UV-textured meshes (SuGaR), MeshSplats significantly outperformed them in rendering quality (PSNR: 28.08 vs. 21.24 for 1M vertices), proving that direct color parameterization is more effective than UV mapping for this task.

• Qualitative Performance:

  • Visual Fidelity: Renders produced in Blender and Nvdiffrast are visually nearly identical to 3DGS without spherical harmonics.
  • Advanced Effects: The method successfully demonstrates real-time reflections, shadows, and physical interactions (e.g., water simulation) in Blender, which are unachievable with standard GS.
  • Limitations: Minor artifacts (fragmented geometry) can occur in large, low-texture areas (e.g., sky) where mesh interpolation struggles to mimic the smooth fall-off of Gaussians. High-frequency details (e.g., dense grass) may be slightly coarser than in pure 3DGS.

5. Significance

MeshSplats bridges a critical gap in the 3D graphics pipeline:

• Bridging Rasterization and Ray Tracing: It allows the high-quality, fast training of GS to be combined with the physical accuracy of ray tracing.
• Workflow Integration: By converting GS to standard meshes, it enables the use of industry-standard tools (Blender, Unreal Engine) for editing, animation, and simulation, which were previously inaccessible to GS models.
• Future Applications: The method opens the door for advanced applications such as interactive editing, physics simulations, and high-fidelity rendering in virtual production and gaming, leveraging the best of both Gaussian Splatting and classical mesh-based rendering.