ZeroSplat: Generalized Referring Segmentation in 3D Gaussian Splatting
The paper introduces ZeroSplat, a training-free and zero-feature framework that addresses the limitations of existing methods by enabling the generalized referring segmentation of arbitrary numbers of targets in 3D Gaussian Splatting through the lifting of 2D Vision-Language Model priors into 3D space via multi-view geometric constraints.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are holding a magic camera that doesn't just take a flat photo, but captures a whole room in 3D, like a cloud of tiny, glowing dust motes floating in space. This is the world of 3D Gaussian Splatting, a new way computers build 3D scenes that are so sharp and fast they can be viewed in real-time. But for a long time, these 3D scenes were "blind" to language; you couldn't tell the computer, "Show me the red chair," and have it understand what you meant. Researchers tried to fix this by teaching the computer to read text and point at things, but they hit a wall: the old methods were like a strict librarian who only knew how to find one specific book at a time. If you asked for "all the red chairs" or "no chairs at all," the librarian would get confused or crash. This paper steps into that gap, asking a simple but powerful question: Can we teach our 3D scene to understand complex, messy human instructions that might ask for zero, one, or a whole bunch of things at once?
The authors of this paper, who call their new tool ZeroSplat, say the answer is yes, and they did it without the heavy, expensive training usually required. They argue that the old ways of doing this were fundamentally broken because they tried to understand 3D objects by looking at 2D pictures of them, like trying to understand a sculpture by only looking at its shadow. This made the computer slow and memory-hungry, needing to store massive amounts of data for every single scene. ZeroSplat flips the script. Instead of training a new brain for every room, it uses a clever trick: it takes a smart, pre-trained "Vision-Language Model" (a super-smart AI that already knows how to match words to images) and projects its understanding directly onto the 3D dust motes using geometry.
Think of it like this: Imagine you have a room full of floating balloons (the 3D scene). You want to find the "blue balloons." The old method was like painting a label on every single balloon after spending hours studying the room. ZeroSplat is different. It's like having a smart friend who looks at a few photos of the room, figures out where the blue balloons are, and then uses a laser pointer to tag the actual floating balloons in the 3D space instantly. If you ask for "no balloons," the friend just says, "Okay, nothing to tag." If you ask for "the blue and the red ones," they tag both. The best part? This friend doesn't need to memorize the room beforehand; they just use their existing smarts and the geometry of the room to figure it out on the fly.
The paper introduces a new challenge called Generalized Referring 3D Gaussian Splatting (GR3DGS). This is a fancy name for a simple idea: a system that can handle any number of targets. It can find zero items (if you ask for something that isn't there), one item, or a whole group of items, all based on a sentence you type. To prove this works, the team built two new test sets, GR-LERF and GR-ScanNet, which are like obstacle courses filled with tricky instructions to see if the computer can really understand what "all the red chairs" or "the empty space" means.
When they tested ZeroSplat, the results were impressive. In these tests, ZeroSplat significantly outperformed the previous best methods. While other systems struggled to separate multiple objects or got confused by complex sentences, ZeroSplat managed to pinpoint the exact 3D points it needed to find. For example, on one test set, it achieved a score of 50.8 (measured by mIoU), while the next best method only reached 23.8. On another test, it hit 41.2, beating the runner-up which scored 24.5. The paper suggests that this success comes from two main tricks: first, using a smart AI to pick the best photos of the scene to analyze (so it doesn't waste time on boring, flat walls), and second, using a "spatial diffusion" process that fills in the gaps, making sure the 3D shape of the object is solid and complete, not just a scattered cloud of points.
Crucially, the paper argues against the idea that you need to spend hours training a model for every new scene or store huge amounts of extra data. They explicitly rule out the need for "per-scene optimization" and "heavy semantic features." Instead, ZeroSplat is "training-free" and "zero-feature," meaning it works immediately without needing to learn the scene first. It's a "plug-and-play" solution that runs efficiently on a single graphics card, using about 10 GB of memory, which is much less than the 20–28 GB required by many other methods.
The authors also tested ZeroSplat on simpler tasks, like just finding one object or finding objects by category, and it still did very well, even beating some methods that were specifically designed for those jobs. This suggests that their approach is not just a one-trick pony but a robust way to understand 3D spaces. However, the paper is careful to note that while the results are strong, they are based on specific datasets and simulations they created. They don't claim to have solved every problem in the universe, but they have shown a clear path forward for making 3D scenes that can truly understand human language, whether you want to find one thing, many things, or nothing at all.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.