GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?
This paper introduces GameXpert-Bench, a comprehensive benchmark suite comprising three tracks (GameGen, GameFix, and GameOpt) that evaluate coding agents across the full game development lifecycle, revealing that while current agents excel at generating playable foundations and implementing explicit requirements, they struggle with defect discovery, runtime behavior verification, and preserving functionality during iterative optimization.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a world where you could describe a game to a computer, and it would not just write the code, but build a fully playable world for you to explore. This is the promise of modern artificial intelligence acting as a coding agent. For years, these systems have learned to write software by following instructions, but game development is a different beast entirely. Unlike a simple calculator or a text editor, a video game is a living, breathing machine where logic, sound, visuals, and player movement must all work together perfectly. If the code for the character's jump is slightly off, the game might crash. If the music does not sync with the action, the experience feels broken. The real test of an intelligent machine is not just whether it can write a line of code, but whether it can build a complete, stable, and fun experience that survives the chaos of actual play.
A team of researchers from Tencent, working across their Hunyuan AI division and Lightspeed Studios, decided to put these coding agents to the ultimate test. They built a rigorous evaluation system called GameXpert-Bench to see how close these artificial minds are to becoming true game developers. Instead of just asking the AI to write a game once and checking if it runs, the researchers watched how the AI handled the entire messy lifecycle of making a game. They looked at three distinct stages: creating a game from scratch, finding and fixing hidden bugs, and improving a working game based on human feedback. Their findings reveal a stark reality: while these agents are surprisingly good at building the skeleton of a game, they struggle significantly when asked to find their own mistakes or keep a game working smoothly as it changes.
The first stage of their test, which they call GameGen, asked the AI to build a complete game from a single sentence of instructions, starting with a blank computer screen and no pre-made tools. The researchers gave the agents 97 different challenges, ranging from simple two-dimensional puzzles to complex three-dimensional adventures. The results showed that the best AI models could indeed create playable games. They managed to build the core mechanics, such as moving a character or collecting items, with high reliability. However, the quality dropped sharply when the tasks required more than just the basics. The agents often failed to create rich, detailed worlds or to ensure that the visual elements, like user interfaces, were perfectly aligned. A common failure was seeing buttons overlap with other graphics or having the game crash because the code for the visuals did not match the code for the movement. The study found that while the AI could build a house, it often forgot to put the doors in the right places or ensure the roof would not leak.
The second stage, GameFix, tested the AI's ability to act as a detective. The researchers took 50 high-quality, human-verified games and secretly injected between 19 and 27 specific bugs into each one. These bugs ranged from broken controls to unbalanced difficulty. The AI was then asked to find and repair these errors. In one version of the test, the researchers told the AI exactly where the problems were. In this scenario, the agents performed reasonably well, fixing the known issues. But in the more difficult version, the AI was only given vague complaints, like "the game feels wrong," and had to discover the bugs on its own. Here, the performance collapsed. The best models could only fix a small fraction of the hidden problems. This revealed a critical gap: the AI is excellent at following a list of instructions but is very poor at looking at a broken system, figuring out what is wrong, and fixing it without being told exactly what to do.
The final stage, GameOpt, simulated a real-world collaboration where a human and an AI work together over several rounds to improve a game. The researchers started with a working game and asked the AI to make specific improvements, such as adjusting the difficulty, changing the art style, or tweaking the sound. They did this for six rounds in a row, with each new request building on the previous changes. The goal was to see if the AI could make the game better without breaking the parts that were already working. The results were mixed. The AI could often follow the new instructions, but it frequently introduced new problems or broke old features while trying to implement the new ones. The study showed that maintaining a stable, playable game while constantly changing its parts is a massive challenge for current technology. The agents often lost track of the game's core logic, leading to versions that were unplayable or felt disjointed.
The overarching conclusion from this extensive testing is that the ability to generate a game from a single prompt is not enough to call an AI a true game developer. The current generation of coding agents is reliable at producing a playable foundation and following explicit instructions, but they are not yet capable of the self-correction and long-term planning required for professional development. They struggle to find their own errors, verify that their changes work in the real world, and preserve the integrity of a game as it evolves. The researchers found that the difference between a model that can write code and one that can build a game lies in this ability to handle the unexpected. Until these systems can discover and fix their own mistakes without constant human guidance, they will remain powerful assistants rather than independent creators. The path forward requires not just smarter code generation, but a deeper understanding of how to keep a complex, interactive system running smoothly through the inevitable changes of development.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.