💻 computer science

Incremental Online Scene Reconstruction by 3D Gaussian Triangulation

This paper proposes an incremental online framework that directly triangulates dense 3D Gaussian primitives into high-fidelity explicit meshes, overcoming the limitations of offline implicit conversions through a novel meshing algorithm, plane-based alignment constraints, and dynamic freezing of optimized regions to achieve superior rendering quality and reconstruction accuracy.

Original authors: Yanjin Zhu, Shaofan Liu, Jianke Zhu

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yanjin Zhu, Shaofan Liu, Jianke Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a 3D model of a room while walking through it, frame by frame, like a video game character. Most old-school methods are like a clumsy construction crew: they wait until they have seen the entire house, then they stop, go back to the beginning, and try to rebuild the whole thing from scratch every time a new brick is added. This is slow, memory-hungry, and impossible for robots that need to make decisions right now.

Other modern methods use "invisible clouds" (called implicit fields) to guess what the room looks like. While they look great on screen, turning those invisible clouds into a solid, walkable 3D mesh is like trying to bake a cake by first turning the batter into a ghost, then hoping the ghost can be turned back into a cake later. It's messy, requires a lot of offline processing, and doesn't work well for things that are constantly changing.

The Big Idea: The "Surfel" Surfboard
The authors of this paper propose a totally different approach. Instead of waiting or using invisible ghosts, they treat every single point in their 3D scene like a tiny, flat, 3D surfboard (they call these "Gaussian surfels"). Think of these surfboards as millions of tiny, flat tiles that float in space, each with a specific color and a specific angle.

Their main finding is that you can take these millions of tiny, flat surfboards and directly triangulate them—stitch them together like a mosaic—to create a solid, watertight 3D mesh instantly, as the data comes in. No need to convert them into invisible fields first. It's like snapping LEGO bricks together as you find them, rather than waiting until you have the whole box to start building.

What They Argue Against
The paper explicitly argues against the idea that you need to process the entire scene at once (offline) to get a good mesh. They also argue against the method of turning optimized 3D Gaussians into an "intermediate implicit field" just to extract a mesh later. They say this extra step is a bottleneck that hinders real-time applications. They also show that simply using standard 3D points isn't enough; you need to force these points to act like flat surfaces (planar constraints) to get a clean mesh.

How It Works (The Magic Tricks)

The "Pull" Trick: To make sure these tiny surfboards line up perfectly to form a smooth wall, the system uses a "plane-based pulling constraint." Imagine a magnet pulling a floating surfboard until it lies perfectly flat against the invisible wall it's supposed to represent. This fixes noisy data and ensures the surfboards align with the actual surface.
The "Freeze" Trick: As you walk through a long hallway, your computer memory would eventually explode if it kept trying to optimize every single tile you've ever seen. So, the system has a clever "freeze" button. Once a section of the room has been seen enough times and is perfectly optimized, the system locks it down. It stops trying to tweak those tiles and only focuses its brainpower on the new areas you are walking into. This keeps the memory usage low (around 2325 MB) and the speed high (10.34 FPS).
The "Snip" Trick: Before building the mesh, the system prunes away any surfboards that are just there for show (transparent ones) or ones that don't match the depth of the scene. It only keeps the "heavy" ones that actually represent the physical object.

The Results: How Sure Are We?
The authors tested this on public datasets like Replica and ScanNet++. They didn't just guess; they measured the results against other top methods like RTG-SLAM, MonoGS, and NICE-SLAM.

Accuracy: On the Replica dataset, their method achieved an average accuracy of 1.34 cm (centimeters), which was better than the next best method (RTG-SLAM at 1.41 cm).
Speed: In terms of mapping speed, they hit 10.34 FPS (frames per second), beating RTG-SLAM (3.65 FPS) and MonoGS (1.48 FPS).
Mesh Extraction Speed: This is where the difference is huge. Converting their Gaussians to a mesh took only 5.34 seconds. Compare that to other methods that use "Marching Cubes" (a standard way to turn clouds into meshes), which took 444.26 seconds for a small subset of the same scene.
Visual Quality: The images they generated had a PSNR (a measure of image quality) of 37.85 on average for the Replica dataset, which was higher than all the other methods they tested.

What They Don't Know (Yet)
The paper is very clear about its limits. They admit their method relies heavily on having depth information (knowing how far away things are). They cannot currently reconstruct parts of the room that they haven't seen yet, and they can't do it using only regular RGB photos (color images) without depth data. They suggest that future work might try to solve this using only RGB, but for now, depth is a requirement.

In short, this paper suggests that by treating 3D points as flat surfboards and stitching them together directly, we can build 3D maps of the world in real-time, with high accuracy, and without running out of memory. It's a shift from "wait and rebuild" to "build as you go."

Technical Summary: Incremental Online Scene Reconstruction by 3D Gaussian Triangulation

Problem Statement
Incremental 3D scene reconstruction is critical for real-world applications such as Augmented Reality and autonomous robotics, where immediate access to partial results is necessary for decision-making. While 3D Gaussian Splatting (3DGS) has demonstrated superior performance in real-time novel view synthesis, existing approaches for mesh extraction from Gaussians face significant limitations. Most current methods require offline conversion of optimized Gaussians into intermediate implicit fields (e.g., Neural SDFs or volumetric grids) before extracting explicit meshes via algorithms like Marching Cubes or Poisson reconstruction. This batch-processing requirement hinders seamless integration with downstream tasks and prevents continuous scene expansion. Furthermore, traditional online methods based on Truncated Signed Distance Fields (TSDF) suffer from fixed voxel resolution and high memory consumption, while neural implicit methods often incur heavy computational loads and struggle with unbounded scenes.

Methodology
The authors propose a novel online framework that incrementally reconstructs and updates high-fidelity explicit meshes by directly triangulating a dense geometric Gaussian representation, bypassing the need for intermediate implicit fields. The core components of the method are:

Dense Geometric Gaussian Representation:
The scene is represented by a set of 3D Gaussian primitives explicitly constrained to act as "Planar Elliptical Surfels." Unlike standard 3DGS optimized for volumetric radiance, this representation enforces:
- Planar Constraint: The third scale component ( $s_3$ ) is forced toward zero, flattening the Gaussian to align with the local tangent plane.
- Opacity Constraint: High opacity ( $\alpha \approx 1$ ) is enforced to approximate hard physical geometry.
- Initialization: Gaussians are initialized from oriented point clouds derived from RGB-D streams, inheriting positions and normals, with tangential scales determined by local point spacing.
Online Mapping and Optimization:
The system incrementally updates the Gaussian set using a dual-branch rendering strategy (photometric and geometric).
- Adding/Pruning: Opaque Gaussians are added to fill depth gaps, while transparent Gaussians are added to enhance color fidelity. Pruning removes primitives that do not contribute significantly to depth or color rendering.
- Geometric Constraints: To ensure mesh readiness, two specific losses are introduced:
  - Plane-based Pulling Loss: Aligns Gaussian centers toward the underlying surface points represented by the oriented point clouds.
  - Normal Consistency Loss: Enforces alignment between the Gaussian normal (approximated by the shortest-axis direction) and the local surface normal.
- Sparse Loss: Encourages Gaussians to become either fully transparent or fully opaque, reducing ambiguity.
Gaussian Triangulation and Meshing:
Instead of converting Gaussians to an implicit field, the authors propose a direct meshing algorithm:
- Geometric Gaussian Set Selection: A subset of Gaussians is selected based on high opacity ( $\alpha > 0.9$ ) and depth fidelity (matching the reference depth within a threshold $\tau$ ).
- Fast Triangulation: Using a compressed octree, valid neighbors are identified based on mutual visibility and normal consistency. A weighted fusion refines the Gaussian positions. An angle-based greedy algorithm then connects the central Gaussian with angularly sorted neighbors to form triangles, preventing degenerate sliver triangles.
- Local Remeshing: To handle topological conflicts during incremental fusion, the system performs edge splitting, collapsing, and flipping, followed by Laplacian smoothing.
Memory Efficiency via Freezing:
To prevent unbounded resource growth during long-sequence scanning, the framework dynamically freezes regions that have been fully optimized. A region is frozen when its cumulative observation count exceeds a threshold and its optimization loss has stably converged. Frozen regions are removed from the active optimization graph, significantly reducing memory and computational overhead.

Key Contributions

Novel Triangulation Algorithm: A direct triangulation method on the Gaussian set that enables fast mesh extraction and incremental updates from Gaussian surfels without offline post-processing.
Plane-based Pulling Constraint: A geometric constraint that guides Gaussian surfels to form a coherent structure, rectifying input noise and faithfully representing underlying surfaces.
Memory-Efficient Incremental Framework: A strategy that progressively freezes optimized mesh regions, allowing for scalable, continuous surface reconstruction with reduced memory footprint.

Experimental Results
The method was evaluated on the Replica and ScanNet++ datasets, comparing against volumetric TSDF methods (KinectFusion), NeRF-based SLAM (NICE-SLAM, Point-SLAM), and Gaussian-based approaches (RTG-SLAM, MonoGS).

Reconstruction Accuracy: The proposed method achieved the highest accuracy (lowest error in cm) and accuracy ratio on the Replica dataset, outperforming RTG-SLAM and MonoGS. It produced finer geometric details and better preserved planar surfaces.
Rendering Quality: In terms of PSNR, SSIM, and LPIPS, the method demonstrated superior rendering performance across almost all sequences compared to baselines, reducing artifacts and preserving fine-grained details.
Efficiency: The system achieved a mapping speed of 10.34 FPS with a peak memory usage of 2325 MB on the Replica 'Office0' sequence, significantly outperforming MonoGS and RTG-SLAM in both speed and memory efficiency.
Mesh Extraction Speed: The direct triangulation approach reduced mesh extraction time to 5.34 seconds, compared to hundreds of seconds for methods relying on Marching Cubes or Poisson Surface Reconstruction on implicit fields.

Significance
The paper claims that this work addresses the fundamental bottleneck of offline processing in high-fidelity 3D reconstruction. By enabling the direct extraction of watertight meshes from optimized 3D Gaussians in an online, incremental manner, the method supports seamless integration with downstream tasks and continuous scene expansion. The approach demonstrates that explicit Gaussian representations can be effectively utilized for both high-quality rendering and accurate surface reconstruction without the computational and memory costs associated with intermediate implicit fields. The authors note a limitation in their reliance on depth information for unobserved regions and suggest future work in RGB-only reconstruction.

Technical Summary: Incremental Online Scene Reconstruction by 3D Gaussian Triangulation

More like this