WorldRover: A Scalable Synthetic Video Data Engine for World Exploration with Rich Annotations
This paper introduces WorldRover, a scalable data engine built on Unreal Engine that generates long-range, richly annotated synthetic video sequences of artist-built environments, providing synchronized RGB, metric depth, camera trajectories, and diverse viewpoint data to train models for coherent world exploration and reconstruction.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
To understand how a machine might learn to navigate a world, we must first understand what it means to see that world. For a computer to move through a space, it needs more than just a picture of what is in front of it. It needs to know how far away objects are, how the camera itself is moving, and how the shapes of things change as they pass behind one another. In the real world, capturing all of this information at once is incredibly difficult. A camera can record a video, but it cannot automatically tell you the exact distance to a tree or the precise path the person holding the camera walked. Scientists often have to guess these details after the fact, and those guesses can be wrong, especially when objects block the view of one another. Without perfect knowledge of the geometry and the motion, it is hard to teach a computer to build a stable, coherent mental map of a place that it can explore and return to.
This is the challenge that a team of researchers at Alaya Lab, the University of Tokyo, and the Shanghai Innovation Institute has tackled with a new system called WorldRover. Instead of trying to extract perfect data from messy real-world videos, they built a machine that generates the data from scratch. They created a pipeline that takes artist-built virtual environments and sends a camera on long, minute-scale journeys through them. The key innovation is that this system does not just record the video; it records the entire history of the journey simultaneously. As the camera moves, the system captures the exact path it took, the precise distance to every point in the scene, and how every pixel in the image relates to the one before it. Because the world is synthetic, the researchers know the truth of every frame: they know exactly where the camera was, how the light hit the walls, and where every object is located in three-dimensional space.
The result of this effort is a massive collection of data named WorldRover-10M. It contains over six thousand distinct video sequences drawn from thirty-two different virtual environments, ranging from city streets and subway stations to forests and cathedrals. In total, the dataset holds more than twenty-one million frames of video, amounting to over two hundred hours of footage. What makes this collection unique is that it offers the same journey seen from three different perspectives: a first-person view as if walking through the world, a third-person view following a character from behind, and a 360-degree panoramic view that captures everything around the observer. For every single frame, the system provides a matching set of labels. These include a map of the exact depth of the scene, a record of how the camera moved, and a dense tracking of how points in the image shift from one moment to the next.
The researchers designed this engine to solve a specific problem: the need for data where the cause of visual change is never confused. In a normal video, it is hard to tell if a pixel moved because the object moved or because the camera turned. In WorldRover, the researchers can replay the exact same route under different conditions. They can show the same walk through a city in bright daylight, then in heavy fog, or even as a neutral white model where all textures are removed but the shapes remain. Because the path and the geometry stay identical while only the appearance changes, a computer learning from this data can learn to separate the structure of the world from the way it looks. This separation is vital for training models that need to understand a world deeply enough to navigate it or reconstruct it in three dimensions.
To build this, the team used a powerful game engine, a tool typically used to create video games, but they repurposed it for scientific data generation. They prepared thirty-two distinct environments created by artists, ensuring they were rich with detail and clutter, just like real places. They then programmed three different ways to move through these spaces. One method used a pre-calculated map of walkable paths to plan efficient routes. Another used a reactive agent that sensed obstacles as it moved, sometimes wandering or getting stuck, much like a person exploring a new room. The third method allowed a human to record a path manually. These routes were then rendered offline, meaning the computer took its time to calculate every ray of light and shadow with high precision, rather than rushing to produce a real-time image. This process allowed them to generate data that is physically accurate, with lighting that changes correctly as the time of day shifts or the weather turns.
The dataset includes specific subsets tailored for different tasks. The third-person sequences are particularly valuable because they show a character moving independently of the camera. This allows researchers to study how a machine can distinguish between the motion of an observer and the motion of an object in the world. The system also tracks points on the character and the background over long periods, even when those points are hidden behind walls or other objects. The data tells the computer not just where a point is when it is visible, but where it would be if it were still visible, a concept known as amodal tracking. This is crucial for understanding that a car behind a building still exists and has a specific location, even when it cannot be seen.
WorldRover-10M is not just a collection of videos; it is a structured library where every piece of information is tied to the exact moment it was captured. The depth maps are measured in meters, the optical flow records the movement of pixels in precise units, and the camera paths are recorded with high fidelity. The team released this data in a format that allows researchers to easily access the raw information without needing to guess or estimate. They have made the data available to the public, providing a foundation for others to build models that can understand, navigate, and reconstruct explorable worlds. By turning the problem of world exploration into a scalable data-generation task, the researchers have provided a new tool for the field of artificial intelligence, one that offers a level of clarity and control that is impossible to achieve with real-world recordings alone.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.