💻 computer science

Incremental Semantics-Aided Meshing from LiDAR-Inertial Odometry and RGB Direct Label Transfer

This paper proposes an incremental RGB+LiDAR pipeline that leverages vision foundation models for direct label transfer and semantic-aided TSDF fusion to generate high-fidelity, semantically labeled meshes from indoor scans, effectively overcoming geometric drift and point cloud sparsity to outperform state-of-the-art baselines.

Original authors: Muhammad Affan, Ville Lehtola, George Vosselman

Published 2026-04-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Muhammad Affan, Ville Lehtola, George Vosselman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a perfect, 3D digital twin of a massive, ancient cathedral. You have two tools:

A Laser Scanner (LiDAR): It shoots out millions of invisible laser beams to measure distance. It's great at getting the shape right, but it's like a blind person feeling a wall with a cane. It misses the fine details, leaves gaps (holes) in the data, and sometimes gets slightly lost (drift) as it moves through the building.
A Camera (RGB): It sees the world like a human eye. It knows that a red patch is a "brick wall" and a thin gray line is a "railing." It's great at understanding what things are, but it can't tell you exactly how far away they are.

The Problem:
Previous attempts to combine these two were like trying to bake a cake by mixing flour and water separately, then hoping they stick together. The laser scanner would create a bumpy, hole-filled model, and the camera's "knowledge" couldn't fix the laser's mistakes because they weren't talking to each other properly. The result was a digital model that looked blurry, had missing parts, or turned a thin railing into a thick, solid block.

The Solution: "The Smart Architect"
This paper introduces a new method that acts like a Smart Architect who can see the blueprint (the laser data) and read the instructions (the camera labels) at the same time.

Here is how it works, step-by-step, using simple analogies:

1. The "Label Transfer" (The Translator)

Imagine the laser scanner is walking through the room, collecting dots. Every time it stops, the camera takes a picture.

Old way: The camera says, "That's a chair!" but the laser just sees a bunch of dots.
New way: The system acts as a translator. It takes the camera's picture, identifies the "chair," and instantly paints that label onto the specific laser dots that make up the chair. It's like handing a name tag to every single laser dot as it arrives.

2. The "Smart Clay" (TSDF Fusion)

Now, imagine building the 3D model out of a special kind of digital clay (called a TSDF).

The Old Clay: If you told the old system to build a wall, it would use the same amount of clay whether it was building a thin railing or a thick brick wall. This caused thin railings to disappear or walls to look like melted wax.
The New Smart Clay: Because the system now knows what it is building (thanks to the labels), it changes its behavior:
- If it's a thin railing: It uses very little clay and keeps the edges sharp.
- If it's a big wall: It uses more clay to smooth out the bumps and fill in the gaps.
- If it's a floor: It flattens it out perfectly.

This is called "Class-Conditioned Truncation." Think of it as the builder saying, "Oh, this is a delicate vase, I'll be careful! Oh, this is a sturdy table, I can be rougher."

3. The "Double-Check" (Uncertainty Analysis)

The system is also self-aware. It keeps a "worry score" for every part of the model.

Geometric Worry: "I'm not sure about this shape because the laser was far away and the dots are sparse."
Semantic Worry: "I'm not sure about this label because the camera was blurry or the lighting was bad."
By separating these worries, the system knows exactly why a part of the model might be wrong, allowing for better fixes later.

Why Does This Matter?

The authors tested this on real-world datasets (like the Oxford Spires and NTU VIRAL).

The Result: Their model was more accurate (fewer holes, sharper edges) and more complete (it filled in missing parts better) than the current best methods.
The Big Picture: This isn't just about making pretty pictures. It's about creating USD (Universal Scene Description) files. Think of USD as the "PDF of the 3D world." Once you have this perfect, labeled 3D model, you can instantly drop it into video games, VR headsets, or robot simulators.

In a Nutshell:
This paper teaches a computer how to stop guessing. By letting the camera's "eyes" guide the laser scanner's "touch," the system builds 3D models that are not only geometrically perfect but also smart enough to know the difference between a thin railing and a thick wall, resulting in digital twins that are ready for the future of Virtual Reality and robotics.

1. Problem Statement

Reconstructing high-fidelity 3D meshes from LiDAR-inertial scans in large, complex indoor environments (e.g., cultural heritage buildings) faces significant challenges:

Geometric Artifacts: Point cloud sparsity, geometric drift, and fixed fusion parameters often result in holes, over-smoothing, and spurious surfaces, particularly at boundaries between structurally different surfaces (e.g., thin railings vs. broad walls).
Limitations of Existing Methods:
- RGB-D Systems: Rely on dense, close-range, well-textured inputs and struggle in large indoor spaces with varying lighting or textureless surfaces.
- LiDAR-Only Systems: Operate without semantic information, treating all surfaces with fixed fusion parameters (truncation distance, integration weight), which fails to adapt to varying geometric characteristics (e.g., thin structures vs. large planes).
The Gap: There is a lack of systems that transfer Vision Foundation Model (VFM) semantic labels onto sparse LiDAR-inertial odometry maps to actively guide and improve the geometric mesh reconstruction process.

2. Methodology

The authors propose a modular, incremental pipeline that fuses RGB semantic data with LiDAR-inertial geometry. The system operates at the LiDAR frame rate and consists of three main stages:

A. LiDAR-Inertial Odometry (LIO)

Backbone: Uses FAST-LIO2 for tightly coupled fusion of raw LiDAR and IMU measurements.
Strategy: Employs an incremental k-d tree (ikd-Tree) for real-time registration of tens of thousands of points.
Rationale: Uses LIO rather than full SLAM with loop closure to avoid discontinuous pose corrections that would invalidate the incrementally built Truncated Signed Distance Function (TSDF) and require costly re-integration.

B. Direct Label Transfer (RGB to LiDAR)

Semantic Segmentation: Uses OneFormer (a Vision Foundation Model) to generate panoptic segmentation masks for incoming RGB frames.
Projection: Projects 2D masks onto 3D LiDAR points using a dynamic LiDAR-to-camera transform ( $T_{CL}$ ) derived from IMU state propagation.
Filtering: Employs morphological mask erosion, boundary distance rejection, and depth-discontinuity checks to suppress "label bleed" and filter unreliable projections. Ambiguous points are treated as geometry-only observations.

C. Semantics-Aware TSDF Fusion

This is the core innovation, replacing standard fixed-parameter fusion with a label-aware scheme:

Per-Voxel Semantic Fusion: Voxels maintain a label histogram. Labels are updated via majority vote with a confidence increment that decays logarithmically with range (accounting for projection uncertainty).
Dynamic Label-Aware Truncation: The truncation distance ( $\mu$ $μ$ ) is no longer fixed. It adapts based on:
- Semantic Class: E.g., thinner truncation for railings to preserve detail; wider truncation for walls to ensure completeness.
- Range: Adjusts for increased noise at longer distances.
Temporal Label Consolidation: Maintains pairwise label co-occurrence counts to merge fragmented labels over time, ensuring stable long-lived identities.
Mesh Extraction: Marching cubes are applied to the TSDF zero-level set. Triangles inherit the semantic label and instance ID from the nearest voxel.
Export: The final mesh is exported as a Universal Scene Description (USD) asset, enabling interoperability with XR engines (e.g., Unreal Engine).

3. Key Contributions

Modular Incremental Pipeline: A system that transfers VFM-based panoptic labels from RGB to sparse LiDAR-inertial maps via ego-motion-compensated projection. It handles sparse sampling and cross-modal temporal misalignment better than dense RGB-D systems.
Label-Aware TSDF Integration: A novel fusion scheme combining:
- Dirichlet-inspired per-voxel semantic fusion.
- Class-conditioned dynamic truncation distances.
- Temporal label consolidation.
- Result: The fusion adapts to class-specific geometric characteristics, improving mesh quality over geometry-only baselines.
Uncertainty Analysis Framework: A method to decompose reconstruction errors into geometric (drift, noise) and semantic (segmentation errors, projection misalignment) sources using per-voxel uncertainty scores.
Scan2USD Pathway: Demonstrates a direct workflow from indoor LiDAR scanning to USD assets for XR and digital modeling.

4. Results

The method was evaluated on the Oxford Spires (quantitative) and NTU VIRAL (qualitative) datasets, comparing against state-of-the-art geometric baselines ImMesh and Voxblox.

Quantitative Performance (Oxford Spires):
- Accuracy (RMSE): Achieved 14.54 cm, outperforming Voxblox (17.24 cm) and ImMesh (17.40 cm).
- Completeness: Achieved 98.55%, significantly higher than Voxblox (97.17%) and ImMesh (95.99%).
- F1 Score: Achieved 98.58%, demonstrating a superior balance of precision and recall.
Qualitative Observations:
- The proposed method successfully recovers fine high-frequency structures (e.g., spires, railings) that ImMesh misses and avoids the over-smoothing and faded textures seen in Voxblox.
- It maintains geometric continuity in occluded and sparse regions where other methods fail.
Uncertainty Analysis:
- Revealed a limited overlap between geometric and semantic uncertainties, confirming that errors often stem from distinct sources (e.g., geometric drift vs. segmentation artifacts).
- The semantic map captures class transitions even where geometry is smooth, while the geometric map highlights depth discontinuities.

5. Significance

Bridging the Gap: This work effectively bridges the gap between sparse LiDAR scanning and rich semantic understanding, proving that semantic guidance can directly enhance geometric fidelity in mesh reconstruction.
Industrial Applicability: By targeting large, complex indoor environments (cultural heritage, stations) and outputting USD assets, the method provides a viable path for creating digital twins and XR content.
Robustness: The system demonstrates robustness in scenarios where RGB-D fails (large distances, poor texture) while overcoming the geometric limitations of pure LiDAR systems.
Future Impact: The proposed uncertainty decomposition framework offers a new diagnostic tool for understanding multi-modal reconstruction failures, guiding future improvements in sensor fusion and SLAM systems.