💻 computer science

FogDrive: A Multi-Modal Synthetic Driving Dataset for Perception under Graded Fog

This paper introduces FogDrive, a rigorously calibrated, multi-modal synthetic dataset featuring synchronized camera, LiDAR, and radar data across four visibility levels and day/night cycles, designed to bridge gaps in existing benchmarks and enable robust evaluation of perception systems under graded fog conditions.

Original authors: Vansh Panwar

Published 2026-07-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vansh Panwar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to drive a car. You wouldn't just show it a video of a sunny day and expect it to handle a blizzard or a thick fog bank, right? This is the world of autonomous driving, where computers use cameras and sensors to "see" the road. But here's the tricky part: when the weather gets bad, like when fog rolls in, those sensors get confused. The camera sees a blurry mess, and the laser sensors (called LiDAR) get blocked by tiny water droplets. Scientists have been trying to build datasets—huge collections of driving data—to teach robots how to handle this. But most real-world data is messy; you can't control exactly how thick the fog is, and you rarely have a "clean" version of the same scene to compare it to. It's like trying to learn how to swim by jumping into a stormy ocean without ever seeing what calm water looks like. To solve this, researchers need a way to simulate fog perfectly, creating pairs of "before" and "after" scenes so they can test if their robots are actually getting smarter or just guessing.

Enter FogDrive, a new, super-organized digital playground created by researchers at the Indian Institute of Technology, Guwahati. Think of FogDrive as a massive, high-tech video game level where the weather is a dial the scientists can turn with perfect precision. Instead of hoping for foggy days in the real world, they used a simulator called CARLA to generate 660 driving scenes (about 133,000 frames of video) across six different virtual towns. The magic trick? Every single scene is captured four times: once in crystal-clear weather, and three times in fog of increasing thickness. They calibrated these fog levels to specific visibility distances: 160 meters, 100 meters, and 50 meters. It's like taking a photo of your living room, then taking three more photos where you slowly fill the room with thicker and thicker cotton candy until you can barely see the sofa.

What makes FogDrive special is that it doesn't just use cameras. It mimics a real self-driving car's brain, capturing data from four cameras, a 360-degree LiDAR (which acts like a laser scanner), a semantic LiDAR (which knows what objects are), and a radar. The researchers simulated the fog using real physics laws: the Koschmieder model for the cameras (which explains why things look hazy) and the Beer-Lambert law for the LiDAR (which explains how lasers get absorbed). They even checked their work, running a quality audit on 8,000 images and finding that their labels for cars were 95.1% precise and had over 99% recall for vehicles within 40 meters. This means the "ground truth" they are teaching the robots is incredibly reliable.

The researchers used this dataset to test two big questions about how to teach robots to drive in the fog. First, they asked: "Should we teach the robot to see through the fog first, and then find the car?" (a method called "defog-then-detect"). Or, "Should we just teach the robot to find the car directly in the foggy mess?" (called "train-on-fog"). In their simulations, they found that training directly on the foggy data was the clear winner. The robots that learned to drive in the fog without trying to "clean" the image first performed much better. In fact, trying to clean the image first often made things worse, or at best, didn't help as much as just learning from the fog.

Second, they tested whether mixing different types of fog during training helped. They found that if you train a robot on a mix of light, moderate, and dense fog, it becomes much more robust than if you only train it on one type. Interestingly, they discovered that simply adding more data didn't help as much as adding variety. A robot trained on a mix of fog densities (even with less total data) could detect cars better than one trained on a huge amount of just one type of fog. However, when it came to pinpointing exactly where the car was (its geometry), having more data in the mix did help tighten those measurements.

One of the most surprising findings was about how we measure success. Usually, when we try to fix a blurry image, we use metrics like PSNR and SSIM to see how close the "fixed" image looks to the original clean one. The researchers found that these metrics are terrible predictors of whether a robot will actually spot a car. A method that made the image look very "clean" and high-quality according to these scores actually failed to help the robot find cars. Conversely, a method that made the image look a bit weird or over-saturated (but high in contrast) actually helped the robot find cars much better. It turns out that for a robot, seeing the edges and shapes of a car is more important than seeing the perfect color of the paint.

In short, FogDrive provides a controlled, physics-based playground that proves two main things: first, it's better to teach robots to drive in the fog directly rather than trying to clean the fog away first; and second, the way we usually judge "good image quality" doesn't actually tell us if a robot will be safe. By offering this perfectly paired, multi-sensor dataset, the authors hope to help engineers build self-driving cars that don't just work on sunny days, but can actually handle the gloomy, foggy reality of the road.

Technical Summary: FogDrive – A Multi-Modal Synthetic Driving Dataset for Perception under Graded Fog

Problem Statement

Robust perception under adverse weather, particularly fog, remains a critical bottleneck for reliable autonomous driving. Existing benchmarks fail to support the systematic evaluation of sensor fusion and robustness strategies due to two primary limitations:

Real-world datasets suffer from uncontrolled data collection, where fog density is uncalibrated, weather conditions are observed at single severity levels, and paired clean-and-foggy counterparts are unavailable. This entanglement of factors (lighting, traffic, sensor noise) makes it impossible to isolate the specific effect of fog severity on perception performance.
Synthetic alternatives either focus strictly on camera-only image restoration (lacking multi-modal alignment) or lack the paired clean-and-foggy structures required to benchmark "defog-then-detect" pipelines against "train-on-fog" approaches. Furthermore, existing large-scale simulators (e.g., SHIFT) often model weather as continuous parameters without providing matched observations for every scene, precluding controlled comparisons.

There is a distinct need for a dataset that provides paired clean and degraded observations, multiple calibrated fog densities, a complete multi-modal sensor suite, and consistent annotations across identical driving scenes to enable controlled ablation studies of perception robustness.

Methodology

The authors introduce FogDrive, a rigorously calibrated, multi-modal synthetic dataset generated using the CARLA simulator.

Dataset Construction

Scale and Diversity: The dataset comprises approximately 660 scenes (133,000 synchronized frames) across six CARLA towns, with a balanced 50:50 day/night split.
Sensor Suite: Each scene captures data from four synchronized cameras (RGB, depth, semantic segmentation), a 64-channel LiDAR, a semantic LiDAR, and a forward-facing radar.
Paired Design: Every scene is released in four perfectly matched variants: one clear-weather reference and three graded fog levels (Light, Moderate, Dense).
Physics-Based Degradation:
- Camera: Fog is modeled using the Koschmieder atmospheric optics model ( $I(x) = J(x)t(x) + A(1 - t(x))$ ), where transmission decays exponentially with scene depth.
- LiDAR: Fog is modeled using the Beer-Lambert law for attenuation ( $P_{rec} \propto P_{emit} \exp(-2\beta r)$ ), followed by resampling to model backscatter and per-point noise.
- Radar: Released without degradation, as 77 GHz automotive radar is empirically insensitive to fog droplets compared to optical sensors.
Calibrated Visibility: The three fog levels correspond to meteorological visibility distances of approximately 160m, 100m, and 50m (defined by the 5% contrast threshold).
Annotations: The dataset includes 2D and 3D bounding boxes for vehicles (passenger cars, trucks, large vehicles). Annotations are generated via a modified CARLA-2DBBox module using a semantic LiDAR visibility filter to ensure consistency across all weather conditions.

Quality Assurance

A quantitative audit against an independent semantic segmentation ground truth (CARLA's seg camera) on 8,000 images validated the annotations. The results show 95.1% precision and >99% recall for vehicles within 40 meters. Recall degrades gracefully at greater distances due to LiDAR point density limits, and precision is slightly affected by multi-level geometry occlusions.

Key Contributions

FogDrive Dataset: A large-scale, multi-modal benchmark with paired clean and foggy observations across three calibrated densities, enabling systematic evaluation of perception under controlled environmental degradation.
Controlled Benchmarking: The paired design allows for direct comparison of competing robustness strategies (e.g., "defog-then-detect" vs. "train-on-fog") and systematic studies of weather-induced distribution shifts.
Annotation Validation: A rigorous, pre-registered audit protocol establishing high-fidelity annotations (95.1% precision, >99% recall within 40m) and quantifying the limitations of the visibility filter.
Empirical Findings: The paper establishes baseline benchmarks using state-of-the-art architectures (TransFusion, BEVFusion, YOLOv8-m) to address two critical questions in perception research.

Experimental Results

Experiment 1: 3D Multi-Modal Detection (Single vs. Mixed Density)

The authors compared training on single fog densities versus mixing densities during training.

Detection Rate: Training with mixed fog densities (Light, Moderate, Dense) consistently outperformed training on a single density for raw detection rate (mAP), even when the total training budget was matched. This suggests that diversity in fog density is more beneficial than volume alone for detection rates.
Geometric Accuracy: Scaling the training mix (increasing the volume of mixed data) significantly improved the geometric accuracy of detections (Mean True Positive and Average Translation Error). Specifically, mixed-density training reduced translation error by approximately 4 cm under moderate fog compared to single-density training.
Conclusion: Mixing fog densities improves detection rates at no extra data cost, while scaling the mix further tightens geometric accuracy, which is critical for downstream motion planning.

Experiment 2: 2D Image Restoration vs. Direct Training

The authors evaluated whether "defog-then-detect" pipelines outperform direct training on foggy data and whether standard image quality metrics predict downstream performance.

Direct Training Superiority: A detector trained directly on foggy data (Pipeline D) approached the clean-data upper bound and consistently outperformed all "defog-then-detect" pipelines (Pipeline C), regardless of the dehazing method used.
Failure of Image Quality Metrics: Standard metrics like PSNR and SSIM proved to be poor predictors of downstream object detection performance.
- Example: The DehazeFormer-S model achieved the highest SSIM but performed worse on detection than DCP (Dark Channel Prior).
- Example: Fine-tuning AOD-Net improved PSNR by ~4.9 dB but yielded negligible gains in detection mAP compared to the baseline, whereas DCP (which had lower PSNR than the foggy baseline) significantly improved detection.
Conclusion: Image quality metrics prioritize pixel-level reconstruction (texture, color) which does not necessarily align with the feature maps required for object detection (contrast, edges). Direct training on degraded observations is the superior strategy; dehazing is only a fallback when retraining is infeasible.

Significance and Claims

The paper claims that FogDrive fills a critical gap in autonomous driving research by providing the first controlled, multi-modal benchmark that simultaneously offers paired clean/foggy observations, calibrated fog densities, and consistent annotations.

The significance of the work lies in its ability to:

Decouple variables: Allow researchers to isolate the specific impact of fog severity on perception without the confounding factors of real-world data collection.
Guide Data-Centric ML: Demonstrate that mixing fog densities during training is a low-cost, high-reward strategy for improving robustness, and that scaling mixed data improves geometric precision.
Re-evaluate Restoration Pipelines: Challenge the assumption that improving image quality (PSNR/SSIM) leads to better detection, showing instead that direct learning from degraded inputs is more effective and that classical contrast-restoration methods (like DCP) can outperform deep learning-based restoration for detection tasks.

The authors position FogDrive as a tool to accelerate robust, multi-modal research, with the dataset, generation framework, and audit protocols to be fully open-sourced.