VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?
The paper introduces VibeWorlding, a unified framework comprising the VWE-BENCH benchmark and the VibeWorlding-Gym RL training environment, to evaluate and enhance multimodal agents' ability to autonomously construct interactive 3D open worlds from user queries, demonstrating that reinforcement learning can enable open-source models to surpass frontier closed-source systems in this complex task.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine standing in an empty digital room and asking a computer to build a bustling city street, a quiet forest clearing, or a futuristic space station, simply by describing what you want to see. This is the promise of "embodied artificial intelligence," a field where machines learn to interact with and construct three-dimensional spaces just as humans do. For years, researchers have tried to teach computers to arrange digital objects like furniture, trees, and buildings based on text instructions. However, most previous attempts worked only with simple, idealized requests, such as "put a chair here." They struggled when faced with the messy, open-ended nature of real human conversation, where a user might ask for a "cozy, slightly cluttered study" with specific lighting and mood. Furthermore, because the tools to build these worlds were scattered and the testing methods were inconsistent, it was difficult to know if an artificial intelligence was truly understanding a request or just guessing.
A team of researchers has now tackled this challenge with a new framework called VibeWorlding. They created a comprehensive system that allows an artificial agent to not only listen to a user's description but to actively plan, search for the right digital objects, place them in a 3D space, and then look at the result to see if it matches the vision. The researchers built a massive testing ground containing over 2,600 high-quality 3D objects, ranging from small props to large buildings, and used these to create hundreds of complex scenes. They then generated thousands of unique user requests, some with precise instructions and others with vague, open-ended desires, to see how well different artificial intelligences could handle the task. The system acts as a rigorous judge, checking not just if the objects are in the right place, but if they make physical sense—ensuring, for example, that a table isn't floating in mid-air or that a tree isn't growing inside a wall.
When the researchers tested the most advanced artificial intelligence models available today, including those from major technology companies, they found a significant gap between current capabilities and the goal of building these worlds automatically. Even the most powerful models, which can write code and answer complex questions, failed to successfully construct the requested scenes more than 60 percent of the time. The primary reason for this failure was not a lack of understanding of the user's words, but a difficulty in the precise physical act of editing the 3D world. The models often understood that a user wanted a bookshelf moved, but they would calculate the distance incorrectly or move the object in the wrong direction, causing it to collide with other items or float unrealistically.
To bridge this gap, the team developed a specialized training method that teaches the artificial intelligence through trial and error, similar to how a child learns to build with blocks. The system lets the agent try to build a scene, checks the result against strict rules about physics and user intent, and then rewards the agent only when it gets it right. This process, known as reinforcement learning, allowed the researchers to train an open-source model that could eventually outperform the best closed-source models available. Their best-trained model, which started with a base of 30 billion parameters, achieved the highest success rate of all the systems tested. It learned to retrieve the correct objects, place them without collisions, and respect the physical constraints of the environment, effectively mastering the "vibe" of the world the user wanted to create.
The study also revealed that while these models are getting better at understanding the big picture, the hardest part remains the fine details of spatial reasoning. The agents still struggle with precise measurements, such as moving an object exactly seven meters forward without overshooting or undershooting. However, the researchers found that the training method significantly improved the agents' ability to reason about 3D space, turning a major weakness into a strength. By releasing their data, tools, and trained models to the public, the team hopes to accelerate the development of systems that can one day build interactive 3D worlds for gaming, simulation, and virtual reality with the same ease and creativity as a human designer.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.