💬 NLP

GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?

This paper introduces GameCraft-Bench, a benchmark of 140 Godot-based tasks designed to evaluate the ability of coding agents to generate complete, playable games end-to-end, revealing that current frontier models struggle significantly with achieving artifact completeness and interactive coherence despite implementing recognizable mechanics.

Original authors: Tongxu Luo, Rongsheng Wang, Jiaxi Bi, Chenming Xu, Zhengyang Tang, Jianlong Chen, Juhao Liang, Ke Ji, Shuqi Guo, Yuhao Du, Fan Bu, Wenyu Du, Xiaotong Zhang, Kyle Li, Shaobo Wang, Linfeng Zhang, Yuxuan

Published 2026-06-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tongxu Luo, Rongsheng Wang, Jiaxi Bi, Chenming Xu, Zhengyang Tang, Jianlong Chen, Juhao Liang, Ke Ji, Shuqi Guo, Yuhao Du, Fan Bu, Wenyu Du, Xiaotong Zhang, Kyle Li, Shaobo Wang, Linfeng Zhang, Yuxuan Liu, Xin Lai, Chenxin Li, Yiduo Guo, Zhexin Zhang, Xinyuan Wang, Tianyi Bai, Ziniu Li, Benyou Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a robot chef and give it a recipe for a complex, multi-course dinner. You don't just want the robot to write the recipe down on a piece of paper; you want it to actually cook the meal, plate it, and serve it so you can take a bite and taste it.

That is essentially what this paper, GameCraft-Bench, is about. It's a test to see if AI coding agents can actually "cook" a playable video game from scratch, rather than just writing the code for one.

Here is a breakdown of the paper's main ideas using simple analogies:

1. The Problem: "The Recipe vs. The Meal"

In the past, when we tested AI on coding, we mostly checked if the code looked correct on paper (like checking if the recipe ingredients were listed right). But for video games, looking at the code isn't enough. A game needs to run, the graphics need to show up, and the player needs to be able to move a character and see what happens.

The authors argue that to truly test an AI's ability to make a game, we need three things:

The Real Kitchen (Engine Grounding): The AI must work in a real game engine (they used Godot, a popular free tool), not a fake or simplified version. It's like asking the chef to use a real stove, not a toy one.
The Full Meal (Artifact Completeness): The AI can't just give you a single ingredient (like a script for a jumping character). It must deliver the whole, packaged game ready to launch. No missing parts.
The Taste Test (Interactive Verification): You can't just look at the finished dish; you have to eat it. The game must be played. The AI's success is judged by actually running the game, pressing buttons, and seeing if the world reacts correctly.

2. The Test: GameCraft-Bench

The researchers built a giant test kitchen called GameCraft-Bench.

The Menu: They created 140 different challenges across 15 types of games (like platformers, racing games, strategy games, and horror games).
The Process:
1. They give the AI a natural language description (e.g., "Make a 2D platformer where you collect coins and avoid enemies").
2. The AI builds the game in the Godot engine.
3. The AI also has to record a "demo trace"—a video script showing exactly how to play the game to prove it works.
4. A computer system "plays" the game using that script, records the video, and then a smart AI judge watches the video to score it.

3. The Results: "Good at Writing, Bad at Cooking"

The results were surprising and a bit humbling for the AI community. Even the smartest, most advanced AI models (like Opus-4.7 and GPT-5.5) struggled.

The Score: The best AI only got about 41% on the test. Most scored below 40%.
What they did well: The AIs were decent at building the "core loop." If you asked for a game where a character jumps, the character could usually jump. They could build the engine parts.
Where they failed: They struggled to put it all together into a complete experience.
- Missing Content: They often made a game that was too short or had no levels to progress through.
- Broken Visuals: The graphics might be there, but they didn't make sense (e.g., the player can't see the enemy, or the UI is broken).
- The "Demo" Gap: Many AIs built a game but forgot to record the "demo trace" required to prove it worked. It's like the chef cooking the meal but forgetting to plate it for the judge.

4. Key Discoveries (The "Why")

The paper dug into why the AIs failed:

Seeing is Believing: The AIs that did the best were the ones that looked at the game screen while they were building it. They treated the visual output like a debugging tool. If the character looked weird, they fixed the code. AIs that just wrote code without looking at the screen made more mistakes.
Busy isn't Better: One AI model (MiMo) wrote a lot of code and ran many commands, but it didn't get a higher score. It was like a chef chopping vegetables furiously but never actually cooking the dish. Doing more work doesn't guarantee a better result if you miss the final step.
Different Skills: Making the "jump" mechanic work (Mechanics) is different from making the game look pretty (Art) or having enough levels (Content). An AI can be good at one but bad at the others. They aren't all linked together.

5. The Conclusion

The paper concludes that while AI is getting very good at writing code, end-to-end game creation is still a massive challenge.

Current AI can build "fragments" of games or simple prototypes, but it cannot yet reliably take a creative idea from a human and turn it into a polished, playable, complete game package all by itself. The gap between "writing code that looks right" and "delivering a system that works" is still very wide.

In short: AI can write the recipe, but it's still learning how to cook the full meal and serve it to the customer.

Technical Summary: GameCraft-Bench

Problem Definition

The paper addresses the challenge of end-to-end game generation, defined as the process where an autonomous agent transforms a natural-language game specification ( $s$ ) into a playable interactive system ( $G$ ) within a specific development and runtime environment ( $E$ ). Unlike traditional coding tasks where success is measured by code correctness or unit tests, game generation requires the production of a coherent, launchable artifact that responds to player inputs in real-time.

The authors argue that existing evaluation methods fail to capture the full complexity of this task. Current benchmarks often focus on:

Web-based games that lack the complexity of real engine environments.
Localized edits to existing projects rather than complete construction.
Static inspection of code or assets, which fails to verify actual gameplay behavior.

To address this, the paper formalizes end-to-end game generation as a mapping $(s, E) \mapsto G$ and identifies three critical desiderata for a valid benchmark:

Engine Grounding: Development must occur within a real game engine with its native documentation, assets, and runtime constraints.
Artifact Completeness: The agent must deliver a complete, launchable game project (including metadata, scenes, scripts, and assets), not just partial code or prototypes.
Interactive Verification: Evaluation must be based on observed gameplay interactions (player actions and system responses) rather than static code analysis.

Methodology: GameCraft-Bench

The authors instantiate these desiderata in GameCraft-Bench, a benchmark designed to evaluate coding agents in the Godot 4 engine.

1. Task Suite

Scale: 140 tasks across 15 distinct game families (e.g., Platformer, Strategy, Roguelike, Visual Novel, Tycoon).
Task Structure: Each task $\tau = (s, E, \rho)$ $τ = (s, E, ρ)$ consists of:
- $s$ : A natural-language specification describing the player fantasy, core loop, and mechanics.
- $E$ : A Godot-based environment providing a workspace, toolchain, and shared assets.
- $\rho$ : A hidden evaluation rubric used by the verifier.

2. Evaluation Pipeline

The benchmark employs a five-stage pipeline (Figure 5):

Task Packaging: Agents receive the specification and environment.
Agent Generation: The agent produces a complete Godot project ( $G$ ) and a set of replayable interaction traces ( $\Pi$ ) containing timed keyboard/mouse events.
Build Gate: The verifier attempts to launch the project. If the build fails or traces are unparsable, the score is zero.
Replay: The verifier replays the traces in a headless Godot instance, recording gameplay videos and sampling frames (2 FPS).
Scoring: A multimodal judge scores the gameplay evidence against the hidden rubric across four categories:
- Core Mechanics (M): Rules, state transitions, and goals.
- Content Depth (D): Progression, variety, and runtime scope.
- Functional Visuals (V): Readability, feedback, and UI stability.
- Art & Presentation (A): Aesthetic coherence and "authored" feel.
- Final Score Formula: $Score = BUILD \times (w_M M + w_D D + w_V V + w_A A)$ , with weights prioritizing Content Depth and Presentation.

3. Engine Selection

Godot was selected for its open-source nature, lightweight setup, native 2D workflow, text-based scene files, and command-line/headless execution capabilities, which are essential for reproducible automated evaluation.

Key Results

The authors evaluated seven frontier coding agent configurations (including Claude Code, Codex, Kimi Code, and Code Buddy) on the full benchmark.

Overall Performance: End-to-end game generation remains highly unsolved. The strongest agent (Claude Code with Opus-4.7) achieved an overall score of 41.46%. Most other agents scored below 40%, with some (e.g., DeepSeek-V4-Pro) scoring as low as 2.15%.
Category Breakdown:
- Mechanics: Agents performed relatively better here (e.g., Opus-4.7 scored 55.34%), often creating recognizable local interaction loops.
- Content & Presentation: Scores dropped significantly for Content Depth and Art/Presentation, indicating agents struggle to assemble mechanics into complete, polished, and coherent systems.
Execution vs. Playability: While many agents achieved high "Build Pass Rates" (launching the game), they frequently failed to provide valid replay traces or failed to realize the intended gameplay within the traces, leading to low final scores.

Diagnostic Findings

Visual Feedback is Critical: Agents that utilized rendered screen inspections (e.g., Kimi-K2.6) showed better debugging capabilities, correcting issues like misaligned UI or broken level layouts that were invisible in source code alone.
Tool Usage $\neq$ Quality: High volumes of tool calls (particularly shell commands) did not correlate with higher scores. Success depended on closing the loop between building, providing demo traces, and ensuring playability.
Partial Factorization: Performance in mechanics, content, and visuals was only moderately correlated. An agent could produce a mechanically functional game that lacked content depth or visual polish.

Significance and Claims

The paper claims that GameCraft-Bench establishes a new standard for evaluating creative software agents by shifting the focus from code correctness to interactive system correctness.

Benchmark Contract: It defines a benchmark contract requiring agents to operate in a real engine, deliver complete artifacts, and be judged via interaction.
Current Limitations: The results demonstrate that current frontier agents are capable of generating "fragments" of game-like software but cannot yet reliably transform natural-language specifications into fully playable, coherent, and visually consistent game experiences.
Evaluation Paradigm: The work argues that evaluating executable creative systems requires interaction-grounded assessment (playability) that goes beyond static code inspection, build success, or visual plausibility alone.

The authors emphasize that while the benchmark is currently limited to 2D Godot games and visual evaluation (excluding audio), it provides a rigorous framework for diagnosing the specific gaps in current agents' ability to construct complex interactive systems.