Game2World Engine: Unlocking In-the-Wild Gameplay Videos for World Model Training
This paper introduces Game2World, a comprehensive framework featuring the G2WEngine for UI asset extraction and the GameCleaner model for mask-free HUD removal, which transforms raw gameplay videos into high-quality, UI-free training data that significantly enhances world model performance.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine trying to teach a computer to understand how the physical world moves and changes. To do this, researchers need vast amounts of video showing how objects interact, how light shifts, and how scenes evolve over time. For years, scientists have looked to video games as a perfect source for this data. Games offer endless, diverse environments where every action has a clear consequence, providing a rich library of visual experiences that are difficult to capture in the real world. However, there is a significant problem with using raw footage from these games. When a person plays a game, the screen is cluttered with digital overlays: health bars, maps, ammo counters, and chat windows. These elements are not part of the game world itself; they are screen-space decorations added for the player's benefit. If a computer learns from videos filled with these overlays, it gets confused, mixing up the rules of the game with the rules of the interface.
A team of researchers has developed a new system to solve this problem, effectively stripping away the digital clutter to reveal the pure game world underneath. They created a framework that can automatically identify these on-screen interfaces, remove them from the video, and even fill in the missing background so the scene looks complete and natural. By doing this, they have transformed millions of hours of messy gameplay footage into clean, high-quality training data. Their work shows that when computers learn from these cleaned-up videos, they become significantly better at understanding how the world works, improving their performance in predicting motion and visual quality by a measurable margin. This breakthrough suggests that the vast, untapped library of internet gameplay videos can finally be used to build more intelligent and capable artificial intelligence systems.
The core of this project is a new data engine called G2WEngine. The researchers realized that to teach a computer about the world, they first had to teach it what isn't the world. They began by creating a detailed classification system for game interfaces, organizing everything from compasses and health bars to chat logs and pop-up notifications into specific categories. Using this system, the engine scans thousands of real gameplay videos from hundreds of different games. It carefully extracts the digital assets—the actual images of the health bars or maps—and turns them into reusable, transparent pieces. This allows the system to understand exactly what these interfaces look like and where they typically appear on the screen.
Once the system understands the interfaces, it works in reverse to create a massive new dataset. It takes clean, interface-free video clips of game worlds and then artificially adds the extracted interfaces back onto them, creating perfect pairs of "before" and "after" footage. This process generates 96,000 synthetic video pairs where the computer knows exactly what the clean world looks like and what the cluttered version looks like. To ensure the system works on real-world data, they also collected over 1,000 clips from actual internet videos, covering 303 different games. This combination of synthetic and real data provides a rigorous testing ground to see if the removal process holds up under pressure.
The researchers then trained a specialized model, which they named GameCleaner, to perform the removal task. Unlike previous methods that required users to manually draw boxes around the elements they wanted to remove, GameCleaner works automatically. It uses a deep understanding of visual context to identify which parts of the image are the game world and which are the digital overlays. When it removes a health bar or a map, it doesn't just erase the pixels; it reconstructs the hidden scenery behind them, ensuring that the background remains consistent and the motion of the characters stays smooth. The model was tested against other video editing tools and proved to be far superior, successfully removing interfaces while preserving the underlying scene with a high degree of accuracy.
The results of this work were tested in a controlled experiment to see if the cleaned data actually made a difference. The researchers trained two identical video-generation models: one on the original, cluttered gameplay videos and another on the newly cleaned versions. The model trained on the clean data performed significantly better. It produced videos with much higher motion quality and better overall visual fidelity. Specifically, the model trained on the cleaned data showed an 18.8 percent improvement in how well it captured motion and a 6.83 percent improvement in its overall performance score. This proves that the digital overlays were not just visual noise; they were actively confusing the learning process, acting as shortcuts that prevented the computer from truly understanding the dynamics of the environment.
The impact of this research extends beyond just making videos look cleaner. By successfully disentangling the interface from the world, the researchers have unlocked a scalable path to creating better world models. These models are the foundation for future artificial intelligence that can navigate complex environments, simulate physical interactions, and even control robots. The ability to process vast amounts of internet gameplay data without the need for expensive, custom-built simulators means that the pool of training data available for these systems is now effectively limitless. The team has made their dataset, code, and models available to the public, inviting other scientists to build upon this foundation. Their work demonstrates that with the right tools, the chaotic, interface-heavy videos of the internet can be transformed into a pristine, structured resource for teaching machines how the world truly moves.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.