Technical Summary: NeuMesh++

Problem Statement

While neural implicit rendering techniques (e.g., NeRF) have achieved significant success in novel view synthesis and 3D scene reconstruction due to their high rendering quality, they fall short in 3D modeling and editing workflows. Existing neural rendering methods for editing are limited to rigid transformations or category-specific operations. They lack the precision and flexibility required by artists, who prefer mesh-based workflows (e.g., in Blender or Maya) that allow for non-rigid geometry deformation, fine-grained texture editing (swapping, filling, painting), and semantic manipulation. Furthermore, existing methods often require extensive manual annotations for fine-grained editing and struggle with local spatial distinguishability (e.g., distinguishing points inside vs. outside a mesh face) when using simple interpolation on mesh vertices.

Methodology

The authors propose NeuMesh++, a novel mesh-based neural radiance representation that encodes the neural implicit field onto a mesh scaffold. This approach bridges the gap between volumetric neural rendering and traditional mesh-based modeling.

Core Representation

The scene is decomposed into a set of local, vertex-bounded radiance fields. Each vertex $v$ on the mesh scaffold stores four distinct learnable parameters:

1. Geometry Code ($l_g$): Encodes local geometry.
2. Texture Code ($l_t$): Encodes local appearance.
3. Semantic Code ($l_s$): Encodes semantic attributes.
4. Modification Color ($c_m$): A view-independent, decoder-free color attribute to enhance texture editing fidelity.

Rendering Pipeline

For a query point $x$ along a camera ray:

1. Local Space Parameterization: The system identifies $K$ nearest mesh vertices. Instead of using global coordinates, it projects $x$ onto the normal of each neighbor to compute a local indicator $h_k = (p_k, w_k)$, where $p_k$ is the projection distance and $w_k$ is the normalized inverse distance. This addresses the spatial distinguishability issue (inside/outside surface) without requiring signed distance fields (SDF).
2. Decoding: The geometry and radiance decoders ($F_G, F_R$) take the local codes and indicators as input to predict density $\sigma_k$ and color $c_k$ for each neighbor. The final color includes the modification color: $c_k = F_R(l_t^k, h_k, d, n_k) + c_m^k$.
3. Post-Interpolation: A spatial interpolation strategy fuses the contributions from neighboring vertices to obtain the final density $\sigma$ and color $c$ at point $x$. This creates a "local radiance field" around each vertex, agnostic to mesh topology (including non-watertight meshes).

Editing Capabilities

The disentangled nature of the representation enables several editing functionalities:

• Mesh-Guided Geometry Editing: Since the radiance field is aligned to the mesh, deforming the mesh vertices (e.g., via ARAP or physical simulation) directly deforms the rendered object without retraining.
• Texture Swapping: By aligning source and target meshes (using Umeyama and ARAP), texture codes and modification colors are transferred between vertices based on Euclidean proximity.
• Texture Filling: Uses a tile-based approach in UV space to repeat template textures from a pre-captured model onto a target region.
• Texture Painting: Users paint on a 2D view. A spatial-aware optimization strategy identifies affected modification colors via probing rays and fine-tunes only these specific colors, preventing overfitting to a single view and preserving view-dependent effects.
• Semantic-Guided Editing: Semantic codes are distilled from 2D open-vocabulary segmentation (e.g., DINO, SAM). A propagation mechanism transfers a user's 2D click into a 3D semantic mask, allowing for region selection and editing with a single click.

Key Contributions

1. Novel Mesh-Based Representation: A neural radiance field encoded on mesh vertices with disentangled geometry, texture, and semantic codes, enabling seamless integration with mesh-based workflows.
2. Technical Enhancements for Stability and Fidelity:
   • Local Space Parameterization: A new method using local indicators and post-interpolation to stabilize training without a pre-trained teacher model and improve spatial distinguishability.
   • Learnable Modification Color: A view-independent color parameter on each vertex that improves the fidelity of texture editing, particularly for colors outside the decoder's learned spectrum.
   • Spatial-Aware Optimization: A strategy for texture painting that selectively fine-tunes modification colors to maintain consistency across views.
3. Semantic Propagation: A mechanism to aggregate 2D semantic priors into 3D vertex codes, enabling effortless region selection via single-click interaction.
4. Efficiency: The method achieves real-time rendering (approx. 15 fps) and fast editing (seconds for geometry/texture operations) by using compact latent codes and occupancy-guided sampling.

Results

Extensive experiments were conducted on real-world (DTU) and synthetic (NeRF 360°) datasets.

• Rendering Quality: NeuMesh++ outperforms other editable neural rendering methods (NeuTex, NeuManifold, BakedSDF, NeuMesh) in PSNR, SSIM, and LPIPS. It achieves quality comparable to non-editable state-of-the-art methods like Neuralangelo and DVGO, while offering superior editing capabilities.
• Editing Performance:
  • Geometry: Demonstrated successful non-rigid deformation and physical simulation (e.g., bouncing chairs, swaying plants) without fine-tuning.
  • Texture: Showed superior detail preservation in texture swapping and filling compared to NeuMesh, which often produced blurry or distorted results. The modification color allowed for high-fidelity painting of new colors (e.g., unsaturated greens) that standard decoders could not generate.
  • Semantic: Successfully enabled 3D region selection and editing via single 2D clicks.
• Efficiency: The method renders 800x800 images at 15 fps, significantly faster than Neuralangelo (28s) and NeuMesh (~99s). Editing operations (geometry deformation, texture swapping/filling, painting) take approximately 1-2 seconds.

Significance

The paper claims that NeuMesh++ successfully breaks the barrier between volumetric neural rendering and traditional mesh-based 3D modeling. By encoding neural fields on a mesh scaffold with disentangled codes, it delivers a versatile set of editing functionalities (geometry deformation, texture swapping/filling/painting, and semantic editing) that are both efficient and high-fidelity. The method eliminates the need for extensive manual annotations for semantic editing and avoids the sub-optimal convergence issues of previous mesh-based implicit fields by removing the dependency on pre-trained teacher models. The authors emphasize that this approach provides a practical, artist-friendly workflow for editing neural implicit scenes.