💻 computer science

Feasibility of Indoor Frame-Wise Lidar Semantic Segmentation via Distillation from Visual Foundation Model

This paper demonstrates the feasibility of training a frame-wise indoor LiDAR semantic segmentation model without manual annotations by distilling knowledge from Visual Foundation Models via a 2D-to-3D pipeline, achieving up to 56% mIoU on pseudo-labels and 36% mIoU on real labels.

Original authors: Haiyang Wu, Juan J. Gonzales Torres, George Vosselman, Ville Lehtola

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haiyang Wu, Juan J. Gonzales Torres, George Vosselman, Ville Lehtola

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Teaching a Robot to See Without a Teacher

Imagine you are trying to teach a robot to walk through a house and identify objects (like "chair," "wall," or "floor") just by looking at a laser scan. This is called semantic segmentation.

Usually, to teach a robot, you need a human teacher to sit down and spend hours drawing outlines around every single object in thousands of photos. This is expensive, boring, and slow. In the world of Lidar (lasers that create 3D point clouds), this is even harder because the data looks like a cloud of dots with no texture, making it very hard for humans to label.

The Clever Solution: The "Smart Student" and the "Expert Teacher"

The researchers in this paper came up with a clever trick called Distillation. Think of it like a student-teacher relationship:

The Teacher (The Visual Foundation Model): This is a super-smart AI that has already learned to recognize objects in 2D photos. It's like a seasoned art critic who can look at a picture of a room and instantly say, "That's a sofa, that's a window." It doesn't need to be taught; it already knows everything.
The Student (The Lidar Model): This is the robot's brain that needs to learn to recognize objects in 3D laser scans. It's currently blind to the meaning of the dots.

The Magic Trick: Instead of hiring a human to label the 3D laser scans, the researchers let the Teacher look at the 2D photo taken at the exact same moment as the 3D scan. The Teacher says, "Hey, that pixel is a chair." The researchers then project that answer onto the 3D laser dot that corresponds to that pixel.

The Student then tries to copy the Teacher's logic. It learns: "Oh, when the laser hits a dot that lines up with a 'chair' pixel in the photo, that dot must be a chair too."

Why This is a Big Deal for Indoors

This trick has worked great for self-driving cars on highways (outdoors). But moving it indoors is like trying to teach a fish to climb a tree.

Outdoors: Roads and cars look the same everywhere.
Indoors: Every house is different. One has wooden floors, another has tiles. One has a narrow hallway, another has a wide open living room. The lasers also work differently indoors (some spin fast, some slow).

The researchers asked: "Can we make this 'Smart Student' work inside a messy, complex building without a human teacher?"

How They Did It (The Recipe)

The Ingredients: They grabbed four different datasets of indoor robots moving around buildings. These datasets had both a camera (taking photos) and a Lidar (scanning 3D dots).
The Translation: They built a pipeline to match the 2D photos with the 3D dots, even though the cameras and lasers were different in every dataset.
The Training: They let the "Smart Student" learn from the "Expert Teacher" (using a model called OneFormer or DINOv2) to create Pseudo-Labels. These are "fake" labels generated by the AI itself, acting as a substitute for human labels.
The Test: They tested the student on new rooms.

The Results: Good News, But Not Perfect

The "Fake" Score: When they tested the robot against the AI-generated labels (the Pseudo-Labels), it scored a 56%. This is great! It means the robot learned the general idea of what things are.
The "Real" Score: When they tested it against a tiny amount of actual human labels (which they manually created just for testing), the score dropped to 36%.
- Why the drop? The "Teacher" (the 2D AI) isn't perfect. Sometimes it mistakes a shadow for a wall, or gets confused by a glass table. Since the student copies the teacher, it inherits those mistakes.
The Comparison: Even with a 36% score, the robot was three times better than a standard robot trained on outdoor data and forced to work indoors. It proved that the "Distillation" method works much better than trying to force outdoor knowledge into an indoor world.

The Catch (Limitations)

The "Uncanny Valley" of Data: The robot is very good at recognizing big things like walls and floors (67% accuracy) but struggles with small, messy things like chairs or desks.
Sensor Mismatch: If you train the robot on a robot with a specific camera and laser, and then put it in a building with a different camera and laser, it gets confused. It's like teaching someone to drive a Ferrari and then expecting them to drive a tractor immediately.
No Memory: The robot looks at one frame (one snapshot) at a time. It doesn't remember what it saw a second ago. In a real house, you'd want it to remember, "I just saw a door, so this next dot is probably the doorframe."

The Bottom Line

This paper proves that we don't need humans to label every single 3D scan of a building anymore. We can use a "Smart Teacher" (2D AI) to teach a "Student" (3D Lidar AI) how to see.

While the robot isn't perfect yet (it makes about 1 in 3 mistakes compared to a human), it is a massive leap forward. It turns a task that used to require armies of human labelers into something that can be done automatically, paving the way for robots that can map and understand our homes and offices on the fly.

1. Problem Statement

The Challenge: Frame-wise semantic segmentation of indoor LiDAR scans is critical for real-time applications like semantic SLAM, Scan-to-BIM, and 3D scene understanding. However, acquiring dense, frame-wise ground truth annotations for indoor LiDAR data is prohibitively expensive and time-consuming.
Current Limitations:

Supervised Methods: Rely on large annotated datasets (common in outdoor autonomous driving but scarce indoors) and suffer from significant domain gaps when transferred to indoor scenes.
Scene-Level Segmentation: Requires global reconstruction, making it unsuitable for real-time, online processing.
Visual-Only Approaches: Rely on precise calibration and are limited to the overlapping field of view between cameras and LiDAR.
Existing Distillation: Cross-modal distillation (2D-to-3D) has succeeded in autonomous driving but remains unexplored for indoor frame-wise segmentation due to a lack of benchmarks and the complexity of indoor sensor configurations.

2. Methodology

The authors propose adapting the ScaLR framework (originally designed for outdoor autonomous driving) to indoor environments using a 2D-to-3D cross-modal distillation pipeline.

A. Framework Overview

The system establishes a Teacher-Student relationship:

Teacher: A pre-trained Visual Foundation Model (VFM), specifically DINOv2 or OneFormer, which processes 2D RGB images to generate pixel-level semantic features.
Student: A 3D LiDAR backbone network (WaffleIron) that learns to predict point-wise semantic representations by mimicking the teacher's features.

B. Core Components

Cross-Modal Distillation:
- LiDAR points ( $p_i$ ) are projected onto corresponding image pixels ( $\rho(i)$ ) using intrinsic and extrinsic camera parameters.
- The student extracts 3D feature vectors ( $\tilde{f}_i$ ) and the teacher extracts 2D feature vectors ( $g_{\rho(i)}$ ).
- Loss Function: A cosine-similarity loss minimizes the distance between normalized feature vectors, forcing the 3D network to learn semantic context from the 2D teacher without manual 3D labels.
- Key Adaptation: The pipeline handles heterogeneous indoor sensor setups (e.g., dual-lidar vs. single-lidar, different resolutions) by creating specific data-loading modules for datasets like NTU-VIRAL, TIERS, and M2DGR.
Pseudo-Label Generation:
- To supervise downstream tasks (linear probing and fine-tuning), the authors generate pseudo-labels by projecting the 2D segmentation masks from the VFM (OneFormer) onto the 3D point clouds.
- This allows training on large-scale, unlabeled indoor SLAM datasets.
Training Strategy:
- Phase 1 (Distillation): Unsupervised pre-training of the 3D backbone using the similarity loss against the VFM.
- Phase 2 (Linear Probing): Training only the classification head on the frozen backbone to evaluate feature quality.
- Phase 3 (Fine-Tuning): Full model optimization using the generated pseudo-labels.

3. Key Contributions

Feasibility Demonstration: The first study to demonstrate that cross-modal distillation from VFMs can be successfully adapted for indoor frame-wise LiDAR semantic segmentation, a task previously hindered by a lack of annotated data.
Data Pipeline Development: Created a dedicated data processing pipeline to adapt the ScaLR framework to diverse indoor SLAM datasets (NTU-VIRAL, TIERS, M2DGR, and a custom ITC dataset), handling varying sensor configurations and projection geometries.
Pseudo-Label Validation: Validated the reliability of VFM-generated pseudo-labels by manually annotating a small subset (1,720 frames) of the ITC dataset. This provided a "real-label" benchmark to assess the gap between pseudo and ground-truth evaluation.
Comprehensive Evaluation: Provided the first quantitative results for indoor frame-wise segmentation, establishing a baseline for future research in this domain.

4. Key Results

The experiments were conducted on four indoor datasets. Key findings include:

Performance on Pseudo-Labels:
- The distilled model achieved ~56% mIoU on the ITC dataset when evaluated against pseudo-labels.
- On other datasets (NTU-VIRAL, TIERS), fine-tuned models achieved ~51% mIoU and >85% overall accuracy.
Performance on Real Labels (Ground Truth):
- When evaluated against the manually annotated ITC subset, the model achieved 35.8% mIoU.
- Despite the drop, this significantly outperformed a fully supervised baseline (RandLA-Net pre-trained on S3DIS), which only achieved 10.7% mIoU on the same test set.
Domain Generalization:
- Performance is highly domain-dependent. Models trained on one dataset performed best on that same dataset (diagonal performance in cross-dataset tests).
- However, pre-training on a combined "All Indoor" dataset improved generalization across different indoor scenes.
Robustness to VFM Choice:
- The framework remained stable regardless of whether DINOv2 or OneFormer was used as the teacher, indicating the 3D representation is not overly sensitive to the specific 2D encoder.
Efficiency:
- A 24-layer WaffleIron backbone offered the best trade-off, achieving 15.4 Hz inference speed with 50.9% mIoU, suitable for real-time applications.

5. Significance and Implications

Label-Free Learning: The study proves that high-quality 3D semantic representations can be learned for indoor scenes without manual 3D annotations, leveraging the power of 2D Foundation Models.
Real-Time Applicability: The ability to perform frame-wise segmentation at >15 Hz enables its integration into real-time systems like semantic SLAM and autonomous indoor navigation.
Benchmarking: By providing a small manually annotated dataset and rigorous evaluation, the paper fills a critical gap in the literature, offering a reference point for future indoor LiDAR segmentation research.
Limitations & Future Work: The authors acknowledge that domain gaps (sensor differences, scene layout) and pseudo-label noise (projection misalignment) limit current performance. Future work should focus on standardized indoor benchmarks, uncertainty-aware pseudo-label refinement, and incorporating temporal consistency to improve robustness.

In conclusion, this paper establishes a viable pathway for unsupervised indoor 3D semantic segmentation, demonstrating that distilling knowledge from 2D Visual Foundation Models is a powerful strategy to overcome the scarcity of annotated 3D data.