💻 computer science

Toward Robust LiDAR Semantic Segmentation for Real-World Deployment: Evaluation under Coarse Labels, Adverse Conditions, and Domain Shifts

This paper proposes a structured evaluation protocol to assess the deployment readiness of LiDAR semantic segmentation models by analyzing their performance under coarse safety-aligned labels, eight types of adverse corruptions, and domain shifts, revealing significant gaps between standard benchmark rankings and real-world robustness.

Original authors: Samir Abou Haidar, Alexandre Chariot, Mehdi Darouich, Cyril Joly, Jean-Emmanuel Deschaud

Published 2026-09-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Samir Abou Haidar, Alexandre Chariot, Mehdi Darouich, Cyril Joly, Jean-Emmanuel Deschaud

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Autonomous vehicles and mobile robots navigate the world by building a three-dimensional map of their surroundings, a task they accomplish using sensors called LiDAR. These devices shoot out rapid pulses of light and measure the time it takes for the beams to bounce back, creating a dense cloud of points that represents the shape and position of everything nearby. To make sense of this cloud, the vehicle's computer must perform semantic segmentation: a process of labeling every single point to identify whether it belongs to a road, a pedestrian, a building, or a tree. This understanding is the foundation of safe navigation, allowing the machine to distinguish between a harmless bush and a person crossing the street. For years, researchers have developed increasingly sophisticated computer programs to perform this labeling, testing them on standard datasets that feature clean, clear scans of city streets. However, these standard tests often fail to capture the messy reality of the world, where rain distorts the light, sensors vary between different car models, and the most critical mistakes are not just about missing a detail, but about misidentifying a life-or-death object.

A team of researchers has proposed a new way to judge these systems, one that moves beyond the pristine conditions of laboratory benchmarks to ask whether a model is truly ready for the road. They argue that the current methods of evaluation are too narrow, focusing on fine-grained details that may not matter as much as broad safety categories, and ignoring the fact that real-world sensors degrade and environments change. To address this, they created a unified testing protocol that measures three specific things: how well a system understands broad safety categories, how it holds up when the sensor data is corrupted by weather or hardware flaws, and whether it can recognize objects in a completely new city without having seen that city before. They tested a wide variety of modern computer vision architectures on this protocol, running them on both powerful desktop computers and the smaller, embedded chips that actually power real vehicles.

The researchers found that a high score on a standard test does not guarantee a system is safe or reliable. When they grouped the detailed labels into broader, safety-focused categories—such as merging "car," "truck," and "bus" into a single "vehicle" group—many systems performed better, suggesting that some errors in the standard tests were harmless confusions between similar objects. However, this improvement was not universal; some systems that looked good on fine details actually struggled to correctly identify vulnerable road users like pedestrians and cyclists when viewed through the lens of safety priorities. This revealed a gap where a model might be technically accurate but practically dangerous if it fails to recognize a person as a person.

The study also subjected these systems to eight different types of simulated damage to the sensor data, mimicking real-world problems like fog, rain, snow, motion blur, and sensor malfunctions. The results showed that almost all methods suffered significant performance drops under these conditions, proving that current models are not robust enough for unpredictable weather. The type of damage mattered greatly; while some architectures handled missing data well, others crumbled when the sensor introduced noise or when the physical structure of the scan was altered. Furthermore, the researchers discovered a difficult trade-off: the systems that were most robust against these corruptions often required more computing power, making them slower and less suitable for the real-time processing needed in a moving car.

Perhaps the most revealing test was the domain generalization challenge, where models trained on data from one city were asked to operate in a completely different city with different streets, weather, and even different LiDAR sensors. The performance of nearly every system collapsed in this scenario. Models trained on data from one location failed to recognize objects in another, especially when the sensor hardware itself changed. This suggests that the current generation of artificial intelligence is heavily reliant on the specific patterns of the data it was trained on, rather than learning a universal understanding of the world. The researchers concluded that while these systems are impressive in the lab, they are not yet ready for the real world, as they lack the ability to generalize across different environments and maintain reliability when sensors are imperfect. Their work provides a new, more realistic standard for evaluating these technologies, highlighting that true deployment readiness requires a balance of accuracy, safety awareness, and resilience that no single current method has yet achieved.

Technical Summary: Toward Robust LiDAR Semantic Segmentation for Real-World Deployment

Problem Statement

While recent state-of-the-art methods for LiDAR semantic segmentation achieve strong performance on standard benchmarks like SemanticKITTI and nuScenes, existing evaluation protocols remain insufficient for assessing real-world deployment readiness. Current benchmarks focus primarily on clean, single-domain data with fine-grained label taxonomies. This approach fails to address three critical requirements for autonomous systems:

Safety-Critical Semantics: Fine-grained distinctions (e.g., car vs. truck) may penalize semantically reasonable predictions that do not impact safety decisions.
Adverse Conditions: Models are rarely evaluated under realistic sensor degradations (atmospheric, geometric, and sensor-induced corruptions).
Domain Shift: Performance under cross-domain variability (different sensors, environments, and acquisition conditions) without adaptation is largely unassessed.

Furthermore, no unified protocol currently jointly evaluates semantic relevance, robustness, generalization, and computational efficiency.

Methodology

The authors propose a structured evaluation protocol organized around three complementary dimensions, applied to a representative set of state-of-the-art architectures (point-based, projection-based, sparse convolution-based, and fusion-based). All models were retrained from scratch under consistent training settings to ensure fair comparison.

1. Coarse-Label Evaluation (Safety Assessment)

To align evaluation with autonomous driving safety priorities, the authors adopt the COLA framework. This approach groups fine-grained classes into seven coarse categories: Driveable Ground, Other Ground, Structure, Vehicle, Nature, Living Being, and Object.

Objective: To analyze how label granularity affects performance and to reveal if high fine-grained accuracy masks failures in safety-critical categories (e.g., confusing a pedestrian with a vehicle).
Metric: Coarse-label mean Intersection over Union ( $mIoU_{COLA}$ ).

2. Robustness Evaluation (Adverse Conditions)

Models trained on clean data are evaluated directly on corrupted versions of the datasets (nuScenes-C and SemanticKITTI-C) from the Robo3D benchmark without fine-tuning.

Corruptions: Eight types organized into three groups:
- Atmospheric: Fog, Wet Ground, Snow.
- Geometric: Motion Blur, Beam Missing.
- Sensor: LiDAR Crosstalk, Incomplete Echo, Cross-Sensor.
Metric: Mean Corruption Error ( $mCE$ ), normalized against a baseline (MinkowskiNet).
Efficiency: Inference speed is measured on an embedded NVIDIA Jetson AGX Orin to quantify real-time constraints (targeting >10 FPS for SemanticKITTI and >20 FPS for nuScenes).

3. Domain Generalization Evaluation

The protocol assesses the ability of models to operate in unseen target domains without adaptation.

Protocol: Single-source to single-target transfer. Models are trained on SemanticKITTI or nuScenes and evaluated directly on ParisLuco3D.
Challenge: This setup introduces shifts in geographic environment, acquisition conditions, and, in the SemanticKITTI case, sensor characteristics (HDL-64E vs. HDL-32E).

Key Results

Coarse-Label Performance

Granularity Impact: All methods showed improved $mIoU$ when transitioning from fine-grained to coarse-label evaluation, indicating that many fine-grained errors are intra-class and less critical for safety.
Architecture Differences: Point-based methods (e.g., PTv3) achieved the highest coarse accuracy on nuScenes but suffered from low inference speeds (e.g., 4.1 FPS). Projection-based methods offered high throughput but struggled with safety-critical "Living Being" categories (e.g., FRNet achieved 55.6% on living beings on SemanticKITTI).
Trade-off: Fusion-based methods (e.g., HARP-NeXt) provided a balanced trade-off between accuracy, safety-critical perception, and real-time performance.

Robustness to Corruptions

General Degradation: All methods experienced substantial performance degradation under corruptions, with no single architecture dominating across all categories.
Architectural Characteristics:
- Sparse Convolution-based methods (e.g., Minkowski) showed strong robustness to geometric corruptions due to voxel-based spatial aggregation.
- Projection-based methods showed heterogeneous results; FRNet outperformed others in its category by retaining point-level features, but at a high computational cost.
- Sensor Corruptions remained particularly challenging for all architectures, highlighting limited robustness to sensor-specific artifacts.
Dataset Dependency: Models exhibited higher sensitivity to geometric and sensor degradations on SemanticKITTI (denser point clouds) compared to nuScenes.

Domain Generalization

Performance Drop: Generalization to ParisLuco3D was poor across all methods, with top performers achieving only ~32% $mIoU$ (compared to >70% on source domains).
Sensor Shift Impact: The shift from SemanticKITTI (HDL-64E) to ParisLuco3D (HDL-32E) caused a more severe performance drop than the nuScenes-to-ParisLuco3D shift (identical sensors), emphasizing the difficulty of sensor-invariant generalization.
Architecture Trends: Sparse convolution-based methods (Minkowski, SPVCNN) generally outperformed others. Notably, HARP-NeXt demonstrated better generalization than other projection-based methods, suggesting that specific architectural choices (e.g., depth-wise separable convolutions) can mitigate overfitting to source-domain patterns.

Significance and Claims

The paper claims that clean-benchmark accuracy is a poor predictor of deployment readiness. The proposed protocol exposes concrete gaps between standard benchmark rankings and the actual requirements for autonomous driving:

Safety Relevance: High fine-grained accuracy does not guarantee safety-relevant performance, particularly for vulnerable road users.
Robustness Gap: Current methods overfit to clean data distributions and degrade significantly under realistic sensor corruptions.
Generalization Limit: Domain generalization remains insufficient for reliable deployment without adaptation.

The authors conclude that no single state-of-the-art method currently achieves a strong trade-off between accuracy, robustness, generalization, and computational efficiency. The proposed unified protocol serves as a reference for more practically grounded evaluation, highlighting the need for future model designs that prioritize sensor-invariant architectures and robustness-aware training strategies.