ReRef-3D: A Benchmark for Spatial Referring Expression-Guided 3D Scene Rearrangement
The paper introduces ReRef-3D, a comprehensive benchmark comprising over 33,000 language-guided instructions for 3D scene rearrangement, which evaluates models on their ability to generate physically valid placements that satisfy spatial relations, revealing that current state-of-the-art models struggle significantly with complex relational reasoning and physical constraints.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine standing in a room filled with colorful blocks, cylinders, and spheres scattered across a table. A friend points to a gray ball and says, "Put that next to the block that is to the left of the yellow one." In the human mind, this is a simple, automatic task. We instantly identify the yellow block, find the one to its left, and then slide the gray ball into a spot that feels right next to that second block. We understand that "next to" does not mean a single, precise coordinate, but rather a small zone of acceptable space where the ball can rest without falling off the table or crashing into other objects. This ability to translate vague spatial language into a physical action is a fundamental part of how we interact with the world, yet it remains one of the most difficult challenges for artificial intelligence.
For years, researchers have taught computers to recognize objects in 3D space or to answer questions about them, but getting a machine to actually move an object based on a spoken instruction is a different beast entirely. Most existing tests ask a computer to simply point at an object or describe a scene that is already there. They do not ask the machine to change the scene. To bridge this gap, a team of researchers at the University of Colorado Boulder introduced a new benchmark called ReRef-3D. This is not just a test of whether a computer can understand words, but whether it can take those words and physically rearrange a virtual room to match the request. The researchers built a massive dataset containing nearly 34,000 instructions across hundreds of 3D scenes. These instructions range from simple commands like "put the red block here" to complex chains of logic, such as "place the gray ball next to the block that is to the left of the yellow block that is behind the cyan block."
The core difficulty in this task is that the instructions define a region of valid placements, not a single target point. If a computer places a ball slightly to the left or right of the "perfect" spot, it might still be perfectly correct as long as it satisfies the spatial rules and does not collide with anything. To test this, the researchers did not simply measure how close a computer's guess was to a pre-written answer. Instead, they took the computer's proposed location, placed the object there in the virtual scene, and then re-evaluated the entire room. They checked if the object was actually next to the right neighbor, if it was sitting on the table, and if it was visible. This process ensures that the computer is not just memorizing coordinates but is truly reasoning about the physical consequences of its actions.
The team tested three different types of artificial intelligence models on this benchmark. One was a general-purpose 3D vision-language model, another was a model designed to understand 3D scenes through camera views, and the third was a system specifically built for placing objects. The results revealed a clear hierarchy of ability. The most advanced model, LLaVA-3D, successfully placed objects in a valid spot for about 68 percent of the instructions. The other two models struggled significantly, succeeding only about 32 percent and 22 percent of the time. Even the best performer failed on nearly a third of the tasks, showing that while these systems are getting better, they are far from perfect.
A key finding of the study was that understanding the relationship between objects is often easier for these models than ensuring the placement is physically possible. The models frequently identified the correct target and the correct general direction but failed to account for collisions or the edges of the table. For instance, a model might correctly understand that a ball needs to be "next to" a block but place it in a spot where it would float in the air or clip through another object. The researchers also discovered that the specific wording of the instruction mattered very little. Whether the command was phrased in a rigid, template-like style or in a more natural, conversational way, the models performed almost identically. This suggests that the current limitations are not due to a lack of language flexibility, but rather a fundamental difficulty in visualizing and executing spatial changes.
The study also highlighted which types of instructions were the hardest. Commands involving "nearest" or "between" relationships proved to be the most challenging for all models. Placing an object "between" two others requires the system to understand the space relative to two different anchors simultaneously, a task that confused even the strongest model. Similarly, "nearest" instructions require the object to be closer to the target than any other object in the room, a condition that is easily violated if the model misjudges the distance to a third, unmentioned object. In contrast, instructions asking for an object to be the "farthest" away were much easier to solve, likely because the valid area for such a placement is much larger and more forgiving.
Ultimately, the ReRef-3D benchmark demonstrates that while artificial intelligence is making strides in understanding 3D space, the leap from recognizing a scene to rearranging it is still significant. The best models can handle about two-thirds of these spatial puzzles, but they still stumble on the physical realities of the world. The researchers conclude that the ability to ground language in physical action is not a direct consequence of simply being able to identify objects or answer questions about them. It requires a separate, more complex form of reasoning that current systems are only beginning to master. As these models continue to evolve, the focus is shifting from merely seeing the world to understanding how to move within it, a necessary step toward creating robots that can truly assist humans in our physical environments.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.