A Scene Language Model for Open-Vocabulary Scene Mapping
The paper introduces SceneLM, a vision-language model that maintains a persistent, open-vocabulary 3D scene map as a compact structured text list, achieving competitive localization and retrieval performance with significantly reduced memory usage compared to traditional mapping systems.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Robots that move through the world need a way to remember what they have seen. To navigate a room, open a door, or hand a cup to a person, a machine must build a mental map of its surroundings that persists even after it turns away. For years, engineers have built these maps using complex, multi-step systems. These systems detect objects, calculate their positions in three-dimensional space, and then store vast amounts of visual data—like detailed snapshots or mathematical fingerprints of every surface—to keep the map consistent over time. While effective, these methods are heavy on memory and require specialized software to stitch different views together. The question researchers have been asking is whether a single, intelligent system could learn to do this entire job on its own, using a much simpler way to store information.
A team of researchers has developed a new approach called SceneLM, which attempts to replace these heavy, multi-part systems with a single model that maintains a scene map using only text. Instead of storing complex visual data or feature maps, this system keeps a persistent list of objects, their locations, and short written descriptions. When the robot sees a new image, the model reads its current text-based list and decides what to do: it adds new objects it has just spotted, edits the details of objects it already knows about, or removes items that were mistakenly added or are no longer there. The system learns to perform these updates by studying millions of images that were automatically labeled by other AI tools, creating a training pipeline that does not require humans to manually draw 3D maps.
The researchers found that this text-only method works surprisingly well. In tests involving language-based searches and navigation tasks, the model performed as well as or better than existing systems that rely on dedicated perception and geometry modules. More importantly, the memory required to store the scene was six to twelve times smaller than those traditional systems. Because the representation is so compact, the team was able to run the model on a small computer attached to a four-legged robot, allowing it to map a real-world environment in real time. The robot successfully identified and recorded objects like kettles, trash cans, and light switches, correcting its own mistakes as it moved through a room.
This success suggests that the complex, engineered steps usually required to maintain a 3D map can be learned directly by a vision-language model. The system does not just recognize objects; it learns the logic of map maintenance, understanding when to keep an entry, when to refine it, and when to discard it. The training process relied on an automatic pipeline that generated high-quality labels from images, filtering out poor descriptions and creating a dataset large enough to teach the model these behaviors without human intervention. While the system is slower to process each frame than older methods, the trade-off is a drastically reduced memory footprint and a unified approach that handles perception and memory updates in one step.
The experiments showed that the model could handle the messy reality of a moving robot. In a real-world test on a quadruped robot equipped with a camera and a small onboard computer, the system mapped three different environments, including indoor rooms and an outdoor area. It processed images only when the robot moved a specific distance, ensuring it could keep up with the hardware. The final maps contained the correct objects with accurate positions, demonstrating that the text-based representation was sufficient for navigation and object retrieval. The researchers noted that while the current version requires significant computing power to run, the ability to compress a 3D scene into a simple list of words opens the door for more efficient and capable robots in the future.
By proving that a single model can manage a persistent scene state through simple text operations, this work challenges the idea that complex mapping requires complex, separate components. The results indicate that as vision-language models become more capable, they may naturally absorb the tasks of scene maintenance that are currently handled by specialized software. The study does not claim to have solved every problem in robotics, but it provides strong evidence that a lightweight, text-based memory is a viable and powerful alternative to the heavy, feature-rich maps of the past.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.