AceAction-5M: Hierarchical Goals for Generalist Action Learning Across 3D Game Worlds
This paper introduces AceAction, a large-scale dataset of human gameplay across multiple 3D games that provides hierarchical, temporally aligned vision-language-action annotations to enable generalist agents to learn skill composition and subgoal transitions in dynamic environments.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
To teach a machine how to act in the world, researchers need a massive library of examples showing the connection between what an agent sees, what it wants to achieve, and the specific moves it makes to get there. In the physical world, gathering this kind of data is slow, expensive, and dangerous; it requires robots, human supervisors, and repeated setups that can take years to accumulate. Video games offer a practical alternative. They provide rich, three-dimensional environments where human players perform complex tasks every day, and where every visual scene can be perfectly paired with the exact keyboard and mouse commands used to create it. The challenge for scientists has been that while games generate endless data, turning raw gameplay into a useful lesson for artificial intelligence requires more than just recording video. It demands a way to understand not just the final result, but the step-by-step thinking that led there: how a player decides to move, when they switch from exploring to fighting, and how they break a large goal into smaller, manageable tasks.
A team of researchers has addressed this by creating a new collection of data called AceAction-5M, designed to teach computers how to follow complex instructions across many different 3D video games. Instead of just recording what happened, the researchers organized thousands of hours of human gameplay into a structured hierarchy. They took long, continuous sessions of play and broke them down into a clear chain of command: a main mission, a series of intermediate sub-goals, and the specific, moment-to-moment actions required to reach each one. For example, a single task might be to "cross the corridor and climb the stairs to the red-brick house." The system breaks this down further: first, the player must reach the doorway on the right; then, they must climb the stairs; finally, they approach the house. Each of these steps is linked to the exact visual scene the player saw and the precise sequence of key presses and mouse movements they used to execute the step.
The dataset covers a wide variety of virtual worlds, including popular titles like Fortnite, Minecraft, Counter-Strike, and Euro Truck Simulator 2. In total, the collection includes over 5,000 hours of gameplay from 42 different games, representing more than 5.4 million distinct task instructions and nearly 116 million individual keyboard and mouse actions. To build this, the researchers combined recordings from professional players in their own studios with publicly available match data from online communities. They then used advanced computer vision tools to watch the gameplay and automatically identify when a player's objective changed. If a player was walking down a hall and then stopped to shoot an enemy, the system recognized that the "walking" phase had ended and a "combat" phase had begun. It labeled these shifts as distinct stages, assigning a clear start and end time to every sub-goal.
A crucial part of this work is the ability to link these stages to specific visual landmarks. When a player navigates toward a destination, the system identifies the moment they arrive at a recognizable spot, such as a gate, a ramp, or a specific room, and marks that as a waypoint. This allows the data to show not just that a player moved, but that they moved with a specific destination in mind. The researchers then grouped these waypoints into longer sequences to form complete task descriptions. To ensure the data was accurate, they employed a rigorous two-step checking process. First, artificial intelligence models reviewed every single annotation to verify that the written description matched the video and the recorded controls. Then, a team of eight human reviewers manually inspected hundreds of gameplay trajectories. This human audit confirmed that the descriptions were correct 95.8 percent of the time, ensuring that the link between the language, the vision, and the action was solid.
The result is a dataset that explicitly teaches an artificial agent how to switch between different skills as a situation changes. Unlike previous collections that might only describe the final outcome or provide vague instructions, AceAction-5M shows the machine exactly when to stop one activity and start another. It captures the full range of behaviors needed in a dynamic environment, from navigating a map and scouting for enemies to engaging in combat and recovering after a setback. By providing this level of detail, the researchers have created a foundation for training agents that can understand complex, multi-stage instructions and execute them with the same flexibility as a human player. The work demonstrates that by organizing gameplay into clear, hierarchical goals, it is possible to teach machines not just how to move, but how to think through a sequence of actions to achieve a complex objective.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.