DerainSplat: Feed-Forward Clean 3D Gaussian Splatting from Sparse Rainy Views

Imagine you are trying to build a perfect, 3D hologram of a room using only a few photos taken with a smartphone. This is the exciting world of 3D reconstruction, a field where computers try to understand the shape and look of the real world from flat pictures. Usually, these computer "architects" work best when the photos are crystal clear. But what happens when it's pouring rain? The photos get covered in streaks of water, blurry spots from droplets on the lens, and a hazy gray fog. If you try to build your 3D model with these messy photos, the computer gets confused, and the resulting hologram looks warped and broken. This is a big problem for things like self-driving cars or robots that need to "see" clearly to navigate safely, even when the weather is terrible.

For a long time, scientists had two main ways to fix this. One way was to try to clean up each photo individually before building the 3D model, like wiping a dirty window before looking through it. The other way was to build the model slowly, adjusting it over and over for each specific rainy scene, which is incredibly slow. A newer, faster method called 3D Gaussian Splatting has emerged, which can build a 3D scene in a single, lightning-fast glance. However, this fast method was designed for clear days and collapses when faced with rain. The big question researchers have been asking is: Can we teach a computer to look at a few rainy photos and instantly "see" the clean, dry 3D world underneath, without needing to clean the photos first or wait hours for a result?

Enter DerainSplat, a new approach that says, "Yes, we can!" The researchers behind this work realized that trying to clean the photos first and then build the model was a bad idea because the cleaned photos didn't agree with each other, leading to a messy 3D result. Instead, they built a massive training playground where they taught a computer to understand rain itself. They created a synthetic dataset by taking beautiful, clear 3D scenes and digitally adding rain in four realistic stages: first, they dimmed the lights to mimic an overcast sky; next, they added a depth-dependent haze that gets thicker the further away objects are; then, they painted on realistic rain streaks; and finally, they added water droplets stuck to the camera lens that distort the view.

Using this training data, DerainSplat acts like a super-smart detective that doesn't just wipe the window; it figures out exactly how the rain is messing up the view. It predicts "weather factors"—like a map of where the rain streaks are, how thick the fog is, and where the lens droplets are hiding. It then uses these maps to guide the 3D building process. Think of it like a construction crew that knows exactly which bricks are covered in mud. Instead of trying to scrub the mud off the bricks, they simply ignore the muddy parts and fill in the gaps using clean bricks from other angles, while also dimming the confidence of the muddy spots so they don't mess up the structure.

The results are impressive. When tested on various datasets, including real-world rainy scenes, DerainSplat produced much cleaner and more accurate 3D models than previous methods. It outperformed existing fast methods by a significant margin, improving image quality scores by over 4 dB in some tests, and it did so in a fraction of a second. The paper suggests that by learning to separate the "weather" from the "scene" all at once, rather than in separate steps, we can finally build reliable 3D worlds even when the sky is gray and the ground is wet. This means that in the future, robots and autonomous vehicles might be able to navigate rainy streets just as confidently as they do on sunny ones, seeing the true shape of the world through the storm.