Technical Summary: GameEngineBench

Problem Statement

Current benchmarks for coding agents predominantly focus on general-purpose software engineering tasks, such as isolated function synthesis (e.g., HumanEval, MBPP) or resolving issues in standard repositories (e.g., SWE-bench). These evaluations often fail to capture the complexities of stateful, real-time, interactive systems where correctness depends not only on language semantics but also on runtime behaviors like networking, object lifecycles, engine callbacks, and asset management.

Specifically, there is a lack of benchmarks that evaluate scoped native C++ edits within functioning game engine projects. In such environments, a patch may compile and appear syntactically correct yet fail due to:

• Updating only local state instead of authoritative server state.
• Missing critical object-lifecycle transitions (e.g., destruction, garbage collection).
• Breaking interactions with existing gameplay systems (e.g., replication, UI updates, subsystem initialization).

Game development, particularly using mature engines like Unreal Engine, offers a practical testbed for these challenges because the repositories are executable, stateful, and publicly available, yet they require deep integration with complex runtime architectures.

Methodology

Benchmark Design (GameEngineBench)

The authors introduce GameEngineBench, a benchmark comprising 110 tasks derived from nine real-world, open-source Unreal Engine 5 repositories (e.g., HordeTemplateV2Native, ActionRoguelike, GASShooter).

• Task Structure: Each task provides the model with a buildable starting state, a specific list of editable native C++ files, and a text-based behavior specification.
• Scope: Tasks span diverse areas including gameplay mechanics, multiplayer behavior, AI, animation, UI, persistence, data serialization, XR, and rendering plugins.
• Constraints: Models are restricted to editing native C++ source files. While the projects may utilize non-code assets during execution, the benchmarked edits are strictly code-based.

Evaluation Protocol

The evaluation process is designed to prevent leakage of test logic and to assess behavioral correctness rather than reference similarity.

1. Task Calibration: Before inclusion, tasks undergo a multi-step validation:
   • Test Validation: Ensures the project builds, the reference solution passes, and the starting state fails the intended behaviors.
   • LLM-as-a-Judge Review: A strong coding model attempts the task, and an LLM judge reviews the result against the behavior specification to identify under-specified or over-constrained tests.
2. Execution:
   • Models solve tasks in an isolated temporary workspace.
   • Compilation: The modified project must compile successfully.
   • Runtime Testing: Tests are injected and executed via Unreal's Play-in-Editor (PIE) automation in listen-server mode. This simulates a server-client environment to verify authority, replication, and lifecycle correctness.
3. Judging:
   • Primary Metric: Pass@1 (the fraction of tasks solved by a single attempt).
   • Judge: An LLM judge determines success based on whether the implementation satisfies the requested behavior, considering execution results, assertion failures, and the reference solution. The judge does not merely check for code similarity but evaluates if the runtime behavior is correct.
   • Bias Mitigation: The protocol supports cross-family judging to reduce self-judging bias.

Key Contributions

1. A New Benchmark for Integrated C++ Development: GameEngineBench introduces 110 tasks requiring C++ changes that must integrate into functioning Unreal Engine projects. Unlike previous benchmarks, it evaluates code within a live runtime context involving networking, object lifecycles, and subsystems.
2. Behavior-Centric Evaluation Protocol: The authors define a protocol combining scoped file edits, runtime execution (PIE), and LLM-based judge auditing. This shifts the evaluation metric from "does it compile?" or "does it match the reference?" to "does it behave correctly in the engine?"
3. Analysis of Failure Modes: The paper provides a detailed analysis of why current frontier agents fail, demonstrating that low pass@1 scores are driven by structured runtime and integration errors (e.g., authority mistakes, state synchronization, lifecycle bugs) rather than superficial syntax or compilation errors.

Results

The benchmark was evaluated across 12 different model configurations, including frontier models like Claude 3.5/4 (Fable, Opus, Sonnet), GPT-4.5/5.5, Gemini 3.1 Pro, DeepSeek 4, and Qwen 3.7.

• Performance Ceiling: The benchmark is not saturated. The strongest configuration (Claude Fable-5 with max reasoning effort) achieved a 55.5% pass@1.
• Performance Gap: There is a significant gap between the top performer and others (e.g., GPT-5.5 xhigh at 29.1%, Gemini 3.1 Pro at 18.2%).
• Unsolved Tasks: 31 tasks (28%) remained unsolved by any of the 12 evaluated configurations.
• Complementarity: No single configuration solved all tasks. The sets of solved tasks are complementary; different models succeed on different subsets, but model selection alone cannot cover the entire benchmark.
• Failure Analysis:
  • Compile Failures: Less common among top performers; most failures occur post-compilation.
  • Behavioral Failures: The primary failure mode involves cross-system coordination. Models struggle to synchronize actor identity, replication, server authority, and lifecycle events across multiple engine subsystems.
  • Specific Patterns: Common errors include incorrect authority handling (running logic on clients instead of the server), missing replication hooks, incorrect object lifecycle management (e.g., cleanup timing), and incomplete integration with surrounding game systems.

Significance and Claims

The paper claims that GameEngineBench serves as a concrete measure of coding-agent capability specifically for game-engine development, a domain that has been underrepresented in software engineering benchmarks.

• Beyond Syntax: The results highlight that frontier coding agents struggle with deeply integrated C++ development for real-time interactive software. Success requires understanding not just the code, but the engine's execution model, networking constraints, and state management.
• Value of the Benchmark: By grounding tasks in executable game projects, the benchmark reveals failure modes (runtime integration errors) that are invisible to benchmarks focused on isolated functions or static code review.
• Future Direction: The authors position GameEngineBench as a foundation for more comprehensive evaluations of programming within game engines. They note that while the current set focuses on gameplay-facing C++ systems, future iterations should expand to include audio, performance, security, and platform-specific code to fully represent professional game engineering.

The paper concludes that while current agents show promise, they are not yet reliable for the complex, stateful, and integrated nature of production-level game engine development, underscoring the need for benchmarks that test runtime behavior and system integration.