GS$^{2}$CI: Robust Gaussian Splatting For Snapshot Compressive Imaging via Large Vision Model Priors

Imagine trying to take a high-speed video of a race car zooming past, but your camera is so slow it can only snap one single, blurry picture. Now, imagine that single picture is actually a secret code containing hundreds of different moments of the race, all smashed together. This is the wild world of Snapshot Compressive Imaging (SCI). It's a clever trick where a camera doesn't take a video; instead, it takes one photo that has been "modulated" by a special mask, mixing up time and space into a single snapshot. The challenge? Decoding that one messy photo back into a clear, moving video. Usually, scientists try to do this by guessing what the video frames look like, but if the camera is moving around a 3D scene (like a drone flying around a building), those guesses often fall apart, leaving you with a jumbled, ghostly mess.

Enter 3D Gaussian Splatting (3DGS), a newer, super-fast way to build 3D worlds. Instead of building a solid mesh like a sculptor, 3DGS paints the scene with thousands of tiny, fuzzy, 3D clouds (Gaussians) that look like glitter. When you shine a light (or a camera) through them, they blend together to form a sharp image. It's like building a city out of fog that instantly turns solid when you look at it from the right angle. But even this glitter-city builder struggles when the only clue it has is that one scrambled, compressed photo from the SCI camera. The math gets messy, the fog clouds float in the wrong places, and the camera's path gets lost.

This paper introduces GS2CI, a new method that acts like a super-smart detective to solve this puzzle. The authors realized that trying to guess the 3D shape and the camera's path from just one scrambled photo is too hard, so they brought in some heavy hitters: Large Vision Models (VFMs). Think of these as AI models that have "seen" millions of 3D scenes and know exactly how light, depth, and camera movement usually work. GS2CI uses these AI brains to make a smart first guess about where the camera was and what the 3D scene looks like. Then, it uses a special, custom-made rulebook called OSGR (Opacity-Guided Splitting and Growth Regulation) to tidy up the glitter clouds. This rulebook stops the clouds from growing too big or clumping together in weird ways just to satisfy the math. Finally, it uses a second AI to polish the details, making sure the scene looks sharp and real from every angle. The result? A method that can take that one scrambled photo and reconstruct a high-quality, 3D scene that stays consistent even when the camera moves wildly, outperforming previous methods in both speed and clarity.

The Problem: The "One-Shot" Mystery

Imagine you are a detective trying to reconstruct a crime scene, but you only have one photograph. Worse, that photograph is a "compressed" version where the police took a picture of the scene at 8 different times, mixed them all together with a special filter, and handed you the result. You don't know exactly where the camera was for each of those 8 moments, and you don't know what the scene looked like at any single moment. This is the Snapshot Compressive Imaging (SCI) problem.

Most existing methods try to solve this by guessing the video frame by frame. But if the camera is moving around a 3D object (like a drone flying around a statue), these guesses get confused. They might think the statue is moving when the camera is actually the one moving. They struggle because they lack a good "3D brain" to understand depth and perspective. They are like someone trying to solve a 3D puzzle while blindfolded, relying only on the flat shadows cast on the table.

The Solution: A Two-Step Detective Team

The authors of this paper propose a framework called GS2CI that solves this by combining three powerful tools: a 3D AI brain, a 3D glitter-builder, and a custom rulebook.

Step 1: The Smart Guess (3D VFM Initialization)
Instead of starting from scratch, GS2CI first creates "proxy views." It takes the scrambled photo and the known filter pattern to create a rough, guess-work version of what the 8 different frames might look like. Then, it feeds these rough guesses into a 3D Vision Foundation Model (VFM). You can think of this VFM as an AI that has studied millions of 3D scenes. It looks at the rough guesses and says, "Ah, based on what I've seen before, the camera was probably here, and the object is shaped like this." This gives the system a very strong starting point, rather than a random guess.

Step 2: The Glitter Builder (3D Gaussian Splatting)
Once the AI has a rough idea of the camera path and the scene shape, it switches to 3D Gaussian Splatting. This is the method that builds the scene using thousands of tiny, fuzzy 3D clouds (Gaussians). The system tries to arrange these clouds so that when you look at them from the 8 different angles, they perfectly recreate the scrambled photo.

The Twist: The OSGR Rulebook
Here is where the paper gets really clever. When you try to fit 3D clouds to a scrambled photo, the math can get tricky. The system might try to "satisfy" the math by making a single cloud super opaque (very solid) to cover up a mistake, rather than moving the cloud to the right spot. This creates "opacity peaks" that ruin the 3D structure.

To fix this, the authors invented OSGR (Opacity-Guided Splitting and Growth Regulation).

• Opacity-Guided Splitting: If a cloud gets too "solid" in one spot (an opacity peak), OSGR says, "That's suspicious! Let's split that one big cloud into two smaller, more precise clouds." This forces the system to find the real shape instead of hiding behind a solid blob.
• Growth Regulation: The system also keeps a strict count of how many clouds it can use. It prevents the scene from getting bloated with unnecessary clouds, which keeps the reconstruction fast and stable.

Step 3: The Polish (Auxiliary 2D VFM)
After the main 3D scene is built, the system isn't quite done. It creates new, fake viewpoints (synthesized views) that the camera never actually saw. It then uses a 2D Vision Foundation Model to imagine what those views should look like. It compares its 3D scene to these "dreamed" images and tweaks the details of the clouds to make them look even sharper and more realistic. This is like a final touch-up where an artist adds the fine details to a painting.

The Results: Faster, Sharper, and More Robust

The authors tested GS2CI on many different scenes, from simple objects to complex outdoor environments, and even with very high compression (where 32 different views are smashed into one photo).

• Better Quality: In their tests, GS2CI consistently produced clearer images with fewer errors than previous methods. For example, on a scene called "Vender," it achieved a PSNR (a measure of image quality) of 38.52, beating the next best method which scored 36.40.
• Speed: It is incredibly fast. The entire process, from the scrambled photo to the final 3D scene, took about 68.4 minutes on a standard high-end computer (NVIDIA RTX 4090). In comparison, a previous top method (SCINeRF) took over 12 hours (746.84 minutes) to do the same job.
• Robustness: Even when the camera moved in weird, unpredictable ways or the photo was heavily compressed, GS2CI didn't fall apart. It handled "unbounded" scenes (where the view goes on forever) and complex camera paths much better than its competitors.

What It Doesn't Do (Yet)

The paper is honest about its limits. While it works great for static scenes (where the object isn't moving), it struggles with dynamic scenes where objects are moving independently, like a person running or a bird flying. In tests with moving objects (like a "Bear" or "Flamingo" video), the method didn't perform as well as some other specialized techniques. The authors suggest that in the future, they might need to upgrade the "glitter clouds" to be 4D (adding time as a dimension) to handle moving objects better.

Why This Matters

This paper shows that by combining the "common sense" of large AI models with the speed of 3D Gaussian Splatting, we can solve some of the hardest problems in imaging. It turns a single, scrambled photo into a rich, 3D world that you can walk around in, all while being fast enough to be practical. It's a big step forward for making high-speed, 3D cameras that are small, cheap, and powerful.