GraFT: A Training-Free Framework for Spatial Reasoning in Multimodal Large Language Models via 3D Scene Graphs
GraFT is a training-free framework that enhances spatial reasoning in multimodal large language models by leveraging a 3D scene graph to provide deterministic geometry, allocentric layouts, and visual-attribute grounding, achieving significant performance gains on benchmarks like ScanQA and VSI-Bench without the need for costly fine-tuning or dedicated encoders.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
To navigate the physical world, humans rely on an intuitive sense of space. We know how far away a chair is, whether a door is to our left or right, and how the layout of a room connects to the hallway beyond. This ability to interpret the three-dimensional structure of our surroundings is fundamental to how we interact with everything from furniture to vehicles. In recent years, scientists have taught computers to see and speak using massive artificial intelligence models that process both images and text. These systems, known as multimodal large language models, can describe a picture or answer a question about an object with impressive fluency. However, when asked to perform tasks that require precise spatial reasoning—such as measuring the distance between two items, understanding the full layout of a room from a single viewpoint, or determining exactly how big an object is—these models often struggle. They tend to guess based on patterns in their training data rather than calculating the actual geometry of the scene, leading to errors that would be obvious to a human observer.
A team of researchers at Huawei Technologies has developed a new approach to fix this gap without retraining the artificial intelligence models themselves. They introduced a system called GraFT, which acts as a bridge between the raw visual data a computer sees and the logical reasoning it needs to perform. Instead of forcing the AI to learn spatial rules from scratch, GraFT builds a compact, structured map of the environment, known as a 3D scene graph. This map is not a complex image or a cloud of millions of points, but rather a clean, organized list of objects, their sizes, their positions, and how they relate to one another. Once this map is created, the system uses it to provide the AI with the specific type of evidence it needs for any given question, allowing a frozen, untrained model to solve difficult spatial puzzles with high accuracy.
The core innovation of GraFT lies in how it tailors the information it feeds to the AI based on the specific task at hand. The researchers identified that different questions require different kinds of visual evidence. For questions asking for exact measurements, such as the distance between a table and a sofa, the system bypasses the AI's visual interpretation entirely. Instead, it uses a set of symbolic tools to calculate the answer directly from the precise coordinates stored in the 3D scene graph. This ensures the answer is mathematically exact, derived from the known geometry of the scene rather than a guess. For questions about the overall layout of a room, such as determining which way a person is facing relative to a building, the system generates a custom, top-down view of the scene. Unlike a standard photograph, this view is stripped of all clutter, showing only the relevant objects as simple boxes with their true proportions, making the spatial relationships instantly clear. Finally, for questions about what an object looks like, the system does not show the AI every frame of a video. It uses the geometry of the scene to rank the available camera angles, selecting only the few frames where the object is most clearly visible and centered, and discarding the rest.
The researchers tested this framework on two major benchmarks used to evaluate spatial reasoning in artificial intelligence. In one set of tests involving questions about distances, sizes, and counts, the system improved the performance of a standard AI model by a significant margin, with some metrics showing gains of nearly 60 percent. In another set of tests focused on understanding the layout of rooms and navigating through them, the system allowed a basic, untrained model to outperform not only other general-purpose AI models but also several specialized models that had been heavily trained on spatial data. Remarkably, the system achieved these results without changing a single parameter of the underlying AI model; it simply changed the way information was presented to it. The researchers found that by matching the right type of evidence to the right question, they could unlock spatial reasoning capabilities that were previously locked away in models that were thought to be incapable of such tasks.
The success of GraFT suggests that the limitation of current AI models may not be a lack of intelligence, but rather a mismatch between the data they receive and the nature of the question being asked. By providing a clean, structured representation of the physical world, the system allows the AI to focus on reasoning rather than struggling to interpret raw, noisy visual data. The researchers noted that the accuracy of the system is ultimately limited by the quality of the initial 3D map, but because this map is easy to maintain and update, the framework can be extended to new tasks without the need for expensive retraining. This approach offers a practical path forward for integrating spatial understanding into AI systems, proving that with the right tools and the right perspective, even a frozen model can learn to see the world in three dimensions.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.