💻 computer science

ReaLiTy and LADS: A Unified Framework and Dataset Suite for LiDAR Adaptation Across Sensors and Adverse Weather Conditions

This paper introduces ReaLiTy, a unified physics-informed framework for transforming LiDAR data across sensors and adverse weather conditions, alongside the LADS dataset suite, to enable systematic analysis and robust adaptation in intelligent transportation systems.

Original authors: Vivek Anand, Bharat Lohani, Rakesh Mishra, Gaurav Pandey

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vivek Anand, Bharat Lohani, Rakesh Mishra, Gaurav Pandey

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a self-driving car how to "see" the world using a laser scanner (LiDAR). This scanner shoots out invisible laser beams to build a 3D map of the road, cars, and pedestrians.

The problem is that these scanners are picky. A scanner from one car brand might "see" a tree differently than a scanner from another brand. Furthermore, if it starts raining or snowing, the lasers get scattered, dimmed, or blocked, making the 3D map look fuzzy or broken.

Collecting real-world data for every possible car model and every possible weather condition is impossible. It would take forever and cost a fortune. So, engineers usually use computer simulations to create fake data. But here's the catch: most simulations are like bad photocopies. They look okay, but they don't feel real. They don't capture the specific "glow" of a specific scanner or the messy way snowflakes bounce off a laser beam.

This paper introduces a solution called ReaLiTy and a new dataset called LADS. Here is how it works, explained simply:

1. The Problem: The "Bad Translator"

Think of LiDAR data like a language.

Sensor A speaks "German."
Sensor B speaks "French."
Rain is like a heavy fog that mutes the conversation.

If you try to teach a self-driving car using only "German" data (Sensor A), it will get confused when it meets a "French" car (Sensor B) or when it starts raining. Existing tools are like bad translators; they try to translate the words but lose the accent and the tone, making the car's brain think the world is different than it actually is.

2. The Solution: ReaLiTy (The "Master Chef")

The authors created a framework called ReaLiTy. Think of this as a Master Chef in a kitchen.

The Ingredients (Physics): The Chef doesn't just guess how to cook. They use strict rules of physics (like how light bounces off metal, how far the laser travels, and how raindrops scatter light). This ensures the "meal" is scientifically accurate.
The Taste (Learning): The Chef also uses a "tasting spoon" (AI/Deep Learning). They taste the real-world data (the "French" or "German" dishes) and learn exactly how to adjust the spices (intensity and noise) to make the simulation taste exactly like the real thing.

How it works in two steps:

Changing the Scanner: If you have data from a cheap scanner and want to see what a fancy scanner would see, ReaLiTy takes the raw data and "re-seasons" it to match the fancy scanner's unique style.
Changing the Weather: If you have a clear sunny day, ReaLiTy can "sprinkle" snow or "pour" rain onto the data. But it doesn't just add white dots; it calculates how the snow physically blocks the lasers and how the rain makes the signal weaker, creating a realistic "foggy" 3D map.

3. The Result: LADS (The "Recipe Book")

Once the Chef (ReaLiTy) is done cooking, they need to share the recipes. That's where LADS comes in.

LADS is a massive Recipe Book (a dataset). It contains thousands of "dishes" (3D point clouds) that have been transformed.

It has the same street scene, but now it looks like it was taken by a different car.
It has the same street scene, but now it's covered in snow.
Crucially, every "fake" snow scene matches perfectly with the original "clear" scene, like a "Before and After" photo album. This allows researchers to test their AI on the "Before" and see how well it handles the "After."

4. Why Does This Matter? (The "Driving Test")

The researchers tested this by training a self-driving car's brain (an object detector) on these new "fake snow" datasets.

The Old Way: If you train a car only on sunny days, and then put it in a blizzard, it crashes. It has never seen snow before.
The ReaLiTy Way: They trained the car using their "fake snow" data. When they tested the car on real snow (from a different dataset), it performed much better.

It's like training a pilot in a flight simulator that perfectly mimics a storm. When the pilot gets into a real plane during a real storm, they aren't panicked because the simulator felt exactly like the real thing.

Summary

The Problem: Self-driving cars struggle to adapt to different sensors and bad weather because we don't have enough real data.
The Fix: ReaLiTy is a smart tool that uses physics and AI to turn "sunny day" data into "snowy day" data or "Scanner A" data into "Scanner B" data, making it look and act exactly like reality.
The Gift: LADS is the free library of this transformed data, allowing scientists everywhere to build safer, more robust self-driving cars without needing to wait for a blizzard to happen.

In short, this paper gives self-driving cars a "time machine" to practice for any weather or any car model, ensuring they are ready for the real world.

1. Problem Statement

Autonomous vehicles (AVs) rely heavily on LiDAR for 3D perception, but performance degrades significantly due to domain shifts caused by:

Sensor Variability: Different LiDAR sensors (e.g., different manufacturers, channel counts, beam layouts) produce distinct intensity patterns for the same scene.
Adverse Weather: Rain and snow introduce physical phenomena like attenuation, scattering, backscatter, and point dropouts, altering both the geometry and intensity of LiDAR returns.

Current Limitations:

Existing datasets rarely provide physically consistent, paired observations of the same scene under varying sensors and weather conditions.
Current solutions are fragmented: physics-based simulators lack complex sensor responses, while learning-based methods often lack physical consistency and generalization.
There is a lack of standardized benchmarks for evaluating cross-domain adaptation (sensor-to-sensor and clear-to-adverse weather).

2. Methodology: The ReaLiTy Framework

The authors propose ReaLiTy, a unified, physics-informed learning framework designed to transform LiDAR data from a source domain ( $D_s$ ) to a target domain ( $D_t$ ).

A. Modular Architecture

The framework operates under a shared physics-guided learning paradigm with two distinct but unified branches:

Sensor Adaptation Branch: Focuses on transforming intensity distributions to match target sensor characteristics (e.g., channel layout, reflectance calibration) while preserving the original point cloud geometry.
Weather Adaptation Branch: Handles both geometry and intensity. It uses a physics-based module to simulate weather-induced geometric distortions (dropouts, noise) and a learning module to refine intensity patterns.

B. Core Components

Physics-Guided Inputs: Both branches utilize a "physics-modality stack" as input features: Range ( $R$ ), Incidence Angle ( $\theta$ ), and Material Reflectance ($MR$). This anchors the learning process in physical reality.
Weather Simulation Module: Before learning, a physics-based atmospheric scattering model (based on Mie scattering and Beer-Lambert law) converts clear-weather sweeps into adverse-weather geometries. It simulates particle-beam interactions to create realistic point dropouts and noise.
Learning Model (PICGAN): The core uses a Physics-Informed Cycle-Consistent GAN.
- It employs two generators ( $G_{S2T}, G_{T2S}$ ) and two discriminators for unpaired translation.
- Conditioning: The model is conditioned on the physical cues (range, angle, reflectance) to ensure the generated intensity is physically plausible.

C. Loss Functions

The training objective combines three losses to ensure realism and consistency:

Adversarial Loss ( $L_{adv}$ ): Aligns the generated intensity distribution with the target domain.
Cycle-Consistency Loss ( $L_{cycle}$ ): Ensures invertibility (mapping $S \to T \to S$ recovers the original).
Physics Consistency Loss ( $L_{phy}$ ): Enforces that predicted intensities adhere to physical laws.
- Clear Weather: $I_{phy} = \frac{MR \cdot \cos(\theta)}{R}$
- Adverse Weather: $I_{AW} = I_{phy} \cdot e^{-2\alpha R}$ (incorporating atmospheric attenuation $\alpha$ ).

3. Key Contributions

A. ReaLiTy Framework

A unified system capable of sensor adaptation, weather adaptation, or joint adaptation within a single architecture. It bridges the gap between pure physics simulation and pure data-driven learning.

B. LADS (LiDAR Adaptation Dataset Suite)

A derived dataset suite generated using ReaLiTy, featuring:

One-to-One Correspondence: Transformed point clouds that match the original frames of source datasets (SemanticKITTI, nuScenes, Voxelscape) in file structure, indexing, and calibration.
Physically Consistent Targets: Includes paired data for Snow and Rain conditions, as well as sensor-specific intensity adaptations.
Reproducibility: Designed to allow direct integration into existing pipelines without re-calibration.

C. Benchmarks and Models

The release of pretrained models and standardized evaluation protocols to facilitate reproducible research in LiDAR adaptation.

4. Results and Analysis

Qualitative Results

Sensor Adaptation: Successfully transferred sensor-specific intensity signatures from simulated data to real-world domains (SemanticKITTI, nuScenes) while preserving scene geometry.
Weather Adaptation: Generated realistic snow and rain effects, including characteristic attenuation, reduced intensity, and localized point dropouts, matching real-world observations (CADC and Boreas datasets).

Quantitative Results (Downstream Task: 3D Object Detection)

The authors evaluated the utility of ReaLiTy-generated data by training a PV-RCNN detector on various configurations and testing on real-world snow data (CADC).

Baseline Failure: Training on clear-weather data alone resulted in poor generalization to snow (e.g., Car AP 3D @0.7 dropped to ~6.8% for KITTI).
Synthetic Snow Efficacy: Training on ReaLiTy-generated snow data significantly improved robustness (Car AP 3D @0.7 rose to ~28.4% for KITTI), proving the synthetic data captures critical weather features.
Synergistic Combination: The best performance was achieved by combining ReaLiTy-generated snow with real CADC snow data.
- Example (KITTI): "KITTI-Snow (Ours) + CADC" achieved 39.5% AP, outperforming "KITTI-Clear + CADC" (32.6%) and approaching the "CADC (Real Snow) Only" upper bound (47.9%).
Conclusion: The synthetic data provides complementary diversity (covering object poses and distances) that real-world data alone cannot fully capture, acting synergistically to improve detection robustness.

5. Significance

Unified Approach: Solves the fragmentation in the field by providing a single framework for both sensor and weather adaptation.
Data Scarcity Solution: Addresses the high cost and difficulty of collecting large-scale, paired adverse-weather LiDAR data by providing a physically grounded generation method.
Reproducibility: LADS provides the first systematic, paired dataset suite for LiDAR adaptation, enabling standardized benchmarking across sensors and weather conditions.
Safety Impact: By improving the robustness of perception systems in adverse weather through better training data, the work directly contributes to the safety of autonomous vehicles.

The framework and dataset are available via the project page and integrated with the SimDaaS simulator, offering a scalable foundation for future research in intelligent transportation systems.